Reconstructing Clarion's XML tooling, Part 1
If you want to do XML with Clarion you have a few options. A lot of developers opt for one or more of the available third party offerings. These include:
- Capesoft's XFiles (easy conversion to and from Clarion data structures, with some limitations)
- ThinkData's xmlFuse, a COM wrapper around Microsoft's XML parser
- Robert Paresi's iQ-XML, free and highly regarded but source code is not available
(If I've missed any useful Clarion XML third party products please post a comment below.)
What we sometimes forget is that Clarion itself ships with some extensive XML parsing capabilities. Years ago TopSpeed licensed the CenterPoint XML C++ class library (which is itself, reportedly, built on the open source Expat XML parser) and incorporated a wrapper for that library into the Clarion product line.
DOM and SAX
The Clarion XML toolset includes both DOM and SAX parsers. A DOM parser reads an XML file and creates a corresponding Document Object Model (DOM) in memory. A SAX parser reads an XML file and fires off events as it encounters the different parts of the document (elements, attributes etc.). SAX parsers are generally viewed as better for really large documents, but as I don't deal with really large XML documents I've only ever used DOM parsers.
The Clarion code
Clarion ships with a bunch of code to support the use of the CenterPoint parsers. You can find this code in the following files:
- cpxml.inc
- cpxml.clw
- cpxmlif.inc
There are some additional source files that add support for translating XML to and from Clarion structures, and I'll get to those at some future date.
I'm really interested in this core code, however, because it promises me the kind of low level access to XML documents I want and need.
But there's a problem.
Making sense out of the cpxml* files
The cpxml* files are nearly unreadable, for the simple reason that they contain too much code. So I began by looking for specific clues.
Here's what I get when I search all three files for classes (using Notepad++):
First of all I'd like to know what a UI component (based on the ABC WindowManager class) is doing in a core XML parser library, but I suppose I can let that slip for now. VLB stands for Virtual List Box so it's probably something to do with displaying XML data, perhaps for debugging purposes.
But what are the other classes for? Apart from SAXParserClass none of them seems to have anything obviously to do with XML.
Searching for interfaces, however, tells a different story:
I'm not sure (yet) why the interfaces are duplicated between cpxml.inc and cpxmlif.inc, but clearly this is where the good stuff is. To begin with I'll focus my attention on cpxml.inc.
The Clarion help has this definition of an interface:
An INTERFACE is a structure, which contains the methods (PROCEDUREs) that define the behavior to be implemented by a CLASS. It cannot contain any property declarations. All methods defined within the INTERFACE are implicitly virtual.
I'll assume for the purposes of this discussion that you already have some understanding of interfaces.
Examining the interfaces
If you look at the DOMWriter interface in the source editor you'll see this:
This looks just like a standard Clarion interface. And when I started working with the XML library I was able to use it just like a standard Clarion user interface, but I noticed something odd: I never got any code completion on the interface methods.
As it turns out there's some jiggery-pokery going on here. Scroll all the way to the right in the source editor and you'll see the rest of the story:
Get rid of the extra spaces, replace the line continuation characters with line breaks, and format the code according to the Clarion standard and you'll end up with this:
DOMWriter interface
END
map
module('DOMWriter')
setEncoding PROCEDURE(*DOMWriter, const *CSTRING encoding), name('_setEncoding@8'), pascal, raw
setEncoding PROCEDURE(*DOMWriter, const *CSTRING encoding, bool assumeISO88591), name('_setEncoding2@12'), pascal, raw
getEncoding PROCEDURE(*DOMWriter), *CSTRING, name('_getEncoding@4'), pascal, raw
getLastEncoding PROCEDURE(*DOMWriter), *CSTRING, name('_getLastEncoding@4'), pascal, raw
setFormat PROCEDURE(*DOMWriter, UNSIGNED format), name('_setFormat@8'), pascal, raw
getFormat PROCEDURE(*DOMWriter), UNSIGNED, name('_getFormat@4'), pascal, raw
setNewLine PROCEDURE(*DOMWriter, const *CSTRING newLine), name('_setNewLine@8'), pascal, raw
getNewLine PROCEDURE(*DOMWriter), *CSTRING, name('_getNewLine@4'), pascal, raw
writeNode PROCEDURE(*DOMWriter, *Node pNode), *CSTRING, name('_writeNode@8'), pascal, raw
writeNode PROCEDURE(*DOMWriter, const *CSTRING systemId, *Node pNode),cbool, name('_writeNode2@12'), pascal, raw
end
end
It turns out the interface itself is an empty structure, which is why I never got any code completion. Instead of interface methods there are procedure declarations which take the interface as the first parameter.
As you may already know the first parameter of any class's (or interface's) method is the class (or interface) itself, which is why these functions behave in exactly the same way as if they were interface methods. Each function is a call into the SoftVelocity wrapper around the CenterPoint C++ XML library.
The problem
This technique of marrying Clarion interfaces to the C++ library is tricky and most interesting, but it raises a concern, at least in my mind. In other languages I'm used to modeling XML documents with a set of classes, but here I don't have any classes at all, just interfaces. How do I work with the XML data?
Luckily, I've been down this path before. Close to a decade ago I wrote an article in Clarion Magazine on how to create an XML document using the then-C6 DOM parser. Here's the code, which was pretty much a straight port of some of the C++ example code:
DOMRss PROCEDURE !
DOMImpl &DomImplementation
docType &DocumentType
pDoc &Document
pText &Text
pRootElem &Element
pElement &Element
pCommentElem &Comment
pChannelElem &Element
pItemElement &Element
writer &DomWriter
pCData &CDATASection
encoding cstring('UTF-8')
s cciCStringFactory
CODE
DomImpl &= CreateDomImplementation()
DocType &= null
pDoc &= DomImpl.CreateDocument(s.c(''),s.c('rss'),DocType)
pRootElem &= pDoc.GetDocumentElement()
pRootElem.SetAttribute(s.c('version'),s.c('0.91'))
pCommentElem &= pDoc.CreateComment(s.c('ClarionMag DOM RSS example'))
pDoc.InsertBefore(pCommentElem,pRootElem)
pCommentElem.Release()
pChannelElem &= pDoc.CreateElement(s.c('channel'))
pElement &= pDoc.CreateElement(s.c('title'))
pText &= pDoc.CreateTextNode(s.c('Clarion News'))
pElement.appendChild(pText)
pText.Release()
pChannelElem.AppendChild(pElement)
pElement.Release
pElement &= pDoc.CreateElement(s.c('description'))
pText &= pDoc.CreateTextNode(s.c('News, product announcements, |
& 'and other items of interest to Clarion developers'))
pElement.appendChild(pText)
pText.Release()
pChannelElem.AppendChild(pElement)
pElement.Release
pElement &= pDoc.CreateElement(s.c('language'))
pText &= pDoc.CreateTextNode(s.c('en-us'))
pElement.appendChild(pText)
pText.Release()
pChannelElem.AppendChild(pElement)
pElement.Release
pElement &= pDoc.CreateElement(s.c('link'))
pText &= pDoc.CreateTextNode(s.c('http://www.clarionmag.com'))
pElement.appendChild(pText)
pText.Release()
pChannelElem.AppendChild(pElement)
pElement.Release
pElement &= pDoc.CreateElement(s.c('copyright'))
pText &= pDoc.CreateTextNode(s.c('Copyright 1999-2003 by CoveComm Inc.'))
pElement.appendChild(pText)
pText.Release()
pChannelElem.AppendChild(pElement)
pElement.Release
! Add first <item>
pItemElement &= pDoc.CreateElement(s.c('item'))
pElement &= pDoc.CreateElement(s.c('title'))
pText &= pDoc.CreateTextNode(s.c('RPM Email Survey'))
pElement.appendChild(pText)
pText.Release()
pItemElement.AppendChild(pElement)
pElement.Release
pElement &= pDoc.CreateElement(s.c('link'))
pText &= pDoc.CreateTextNode(s.c('http://www.cwaddons.com/email/'))
pElement.appendChild(pText)
pText.Release()
pItemElement.AppendChild(pElement)
pElement.Release
pElement &= pDoc.CreateElement(s.c('description'))
pText &= pDoc.CreateTextNode(s.c('Lee White is looking for ' |
& 'feedback, from current and prospective RPM users, ' |
& 'about email support in RPM.'))
pElement.appendChild(pText)
pText.Release()
pItemElement.AppendChild(pElement)
pElement.Release
pChannelElem.AppendChild(pItemElement)
pItemElement.Release()
! Add second <item>
pItemElement &= pDoc.CreateElement(s.c('item'))
pElement &= pDoc.CreateElement(s.c('title'))
pText &= pDoc.CreateTextNode(s.c('True Edit-In-Place Template'))
pElement.appendChild(pText)
pText.Release()
pItemElement.AppendChild(pElement)
pElement.Release
pElement &= pDoc.CreateElement(s.c('link'))
pText &= pDoc.CreateTextNode(s.c('http://www.audkus.dk'))
pElement.appendChild(pText)
pText.Release()
pItemElement.AppendChild(pElement)
pElement.Release
pElement &= pDoc.CreateElement(s.c('description'))
pText &= pDoc.CreateTextNode(s.c('This new EIP Template ' |
& 'adds full template support for the Clarion ' |
& 'edit-in-place list box. ABC templates only; ' |
& 'includes source and future updates.'))
pElement.appendChild(pText)
pText.Release()
pCData &= pDoc.CreateCDATASection(s.c('This is some text in a CDATA section'))
pElement.AppendChild(pCData)
PCData.Release()
pItemElement.AppendChild(pElement)
pElement.Release
pChannelElem.AppendChild(pItemElement)
pItemElement.Release()
pRootElem.AppendChild(pChannelElem)
pChannelElem.Release()
Writer &= CreateDOMWriter()
Writer.setFormat(format:reformatted)
if Writer.writeNode(s.c('domrss.xml'),pDoc).
pDoc.Release()
DestroyDomImplementation(DomImpl)
There are a couple of issues with this code. One is that all of my interface references (pText, pElement etc.) have a really ugly "p" prefix. I definitely can't call them Text, Element etc. because those are the actual names of the interfaces and I'll get symbol collisions. Giving the interfaces an appropriate prefix (see below) will solve that problem.
The bigger issue, however, is I'm limited in what I can do with the XML document as it only exists in memory allocated by the C++ library. What if I wanted to validate an XML node, or automatically populate it with some attributes when it was created? I don't have those kind of options.
What I really want is a set of Clarion (not C++) classes that I can use to model an XML document. That way I can add whatever code I need anywhere I need it. Of course I'll still use the existing library to read and write the XML documents by using the existing interfaces and API calls.
Gimme dem classes!
The first thing I do when I start any refactoring project is hit Ctrl-A and Ctrl-I to reformat the code to my liking. It's also something I do regularly while I code, such as after moving or changing a block of code in a way that alters indenting.
The next thing was to come up with a set of actual classes (TYPEd, of course) that model the already-defined interfaces (which themselves are a pretty straightforward model of the DOM specification).
The more classes I write, the more adamant I become about two rules:
- Each class must be in its own .INC/.CLW file pair, and
- Each class (and interface) needs a descriptive, prefixed name
There will always be exceptions, but following these two rules will make any class library much easier to understand.
I contacted Bob Zaunere and asked for and received permission to refactor these classes, and they'll eventually be part of SV's future community repository on GitHub. As much as possible I try to follow the example of .NET when it comes to class naming, but since Clarion doesn't support namespaces I use text to mimic .NET naming. And because these are core SV classes, instead of my usual standard I've used the System_XML_ prefix. It's a little presumptuous I suppose, but as I seem to be the first one in the pool....
I renamed selected interfaces to classes as follows:
Old name | New name |
|---|---|
| Attr | System_XML_Attr |
| CDATASection | System_XML_CDATASection |
| CharacterData | System_XML_CharacterData |
| Comment | System_XML_Comment |
| Document | System_XML_Document |
| Element | System_XML_Element |
| Node | System_XML_Node |
| NodeList | System_XML_NodeList |
| Notation | System_XML_Notation |
| Text | System_XML_Text |
I settled on some new class names as well, but for the most part they don't figure into the first round of refactoring. The only one that's radically different is CStringClass, which is a fairly small class used to manufacture CStrings which are needed when communicating with the CenterPoint XML library.
| Old name | New name |
|---|---|
| CStringClass | System_XML_CString |
String creation is a capability needed in other places than just XML so originally I wanted to give it a more general name, but to avoid introducing any outside dependencies for now I've simply called it System_XML_CString. That will probably change in the near future.
I have a pet peeve about using "Class" in the name of a class. I suppose this harkens back to naming conventions like Hungarian notation, where it was helpful and necessary to be able to infer the data type from the variable name. There's no need for this from a code safety point of view since classes are strongly typed, and the compiler will complain if you try to use a class as something other than a class. For readability there's code completion. So adding "Class" to a class name is arguably just noise. And in fact within the ABC library the term "Class" is added to class names inconsistently.
Creating the class template
I almost never write a class from scratch anymore. Instead I use John Hickey's excellent ClarionLive Class Creator and create my classes from standard class templates. Those aren't templates in the Clarion sense; they're class files that contain the basic structure of the class I want to create.
When I create a class I'm almost always imagining that at some time in the future (when the code is stable) it will be compiled into, and exported from, a DLL. And that means I need some compiler directives for the LINK and DLL attributes. You've probably seen these before in the form of the _ABCDLLMode_ and _ABCLinkMode_ symbols. And if you've ever had to set these manually, for whatever reason, you probably know the disastrous consequences (read: GPF) of getting them wrong.
Here's my class header template file:
include('System_XML_IncludeInAllClassHeaderFiles.inc'),once
System_XML_BaseClass Class,Type,Module('System_XML_BaseClass.CLW')|
,Link('System_XML_BaseClass.CLW',_System_XML_Classes_LinkMode_)|
,Dll(_System_XML_Classes_DllMode_)
Construct Procedure()
Destruct Procedure()
End
Looks painful to type, right? That's why it's in a template file, so I don't have to keep typing it. But if all I do is have it in the class template I'll still need to make sure it gets into the project somehow, and I don't have to have to type it there either. That's something the application (*.tp?) templates usually do for us, but in this case I don't yet have a template, I just have a bunch of source code. So instead I create a standard header file that sets these symbols. Here's System_XML_IncludeInAllClassHeaderFiles.inc:
!----------------------------------------------------------------------------
! While in the development phase default to classes being compiled. Once
! the code is stable and a DLL is provided the following three lines can
! be removed.
!----------------------------------------------------------------------------
omit('***',_Compile_System_XML_Class_Source_)
_Compile_System_XML_Class_Source_ equate(1)
***
OMIT('***',_Compile_System_XML_Class_Source_)
_System_XML_Classes_LinkMode_ equate(0)
_System_XML_Classes_DllMode_ equate(1)
***
COMPILE('***',_Compile_System_XML_Class_Source_)
_System_XML_Classes_LinkMode_ equate(1)
_System_XML_Classes_DllMode_ equate(0)
***
Now instead of two symbols I only need to worry about one symbol: _Compile_System_XML_Class_Source_
And I don't even have to worry about that one, since I've added a bit of code at the top to default _Compile_System_XML_Class_Source_ to 1 so the classes will always be compiled. At some future point when the classes are stable and I'm providing a DLL I can remove the first Omit statement and using the classes from a DLL will be the default (which can be overridden by setting _Compile_System_XML_Class_Source_ to true in the project).
Clear as mud? Good.
Here's the class template for the .CLW file:
Member
Map
End
Include('System_XML_BaseClass.inc'),Once
!include('System_Logger.inc'),once
!dbg System_Logger
System_XML_BaseClass.Construct Procedure()
code
System_XML_BaseClass.Destruct Procedure()
code
There's no real code here yet, but I always put a constructor and destructor in the template because most of the time I end up needing one or both. And they serve as an example of how methods are implemented.
When I create a class using the Class Creator it will replace System_XML_BaseClass with whatever class name I specify.
The XML classes
I'll go into the actual classes and some unit tests next time, but here's the list of source files as it stands now. There are some I haven't yet implemented, and I've also added a few new classes which I'll explain in a future article:
System_XML_Attr.clw System_XML_Attr.inc System_XML_AttrList.clw System_XML_AttrList.inc System_XML_CDataSection.clw System_XML_CDataSection.inc System_XML_CenterPointInterfaces.inc System_XML_CharacterData.clw System_XML_CharacterData.inc System_XML_Comment.clw System_XML_Comment.inc System_XML_CString.clw System_XML_CString.inc System_XML_Document.clw System_XML_Document.inc System_XML_Element.clw System_XML_Element.inc System_XML_IncludeInAllClassHeaderFiles.inc System_XML_Node.clw System_XML_Node.inc System_XML_NodeBuffer.clw System_XML_NodeBuffer.inc System_XML_NodeList.clw System_XML_NodeList.inc System_XML_Notation.clw System_XML_Notation.inc System_XML_Text.clw System_XML_Text.inc System_XML_XPath.inc System_XML_XPathQuery.clw System_XML_XPathQuery.inc
Next time: some unit tests and an explanation of how to read and write basic XML.
And for those of you who just can't wait, here's the source.