by Unlicensed user
In the first article in this series I said that most Clarion developers use embed points the wrong way, and by doing so they make their applications more difficult to maintain, test, debug and document. Almost every Clarion developer has done that; I've done it too. In these articles I intend to show how you can improve your code base by taking the majority of that code out of the embed points.
...
Really you have two options: You can browse it in the embed list or in the embeditor, or you can use a tool to extract just the embeds so you can look at them without the distraction of the generated code.
About TXAs
As far as I know, there's only one way to programmatically extract the embed code from an APP. First, you have to export a TXA, which is an import/export text version of the information contained in an APP file. And second, you to parse the TXA to get the embeds, which can be a bit of a pain as the TXA format isn't documented.
...
Code Block |
---|
?progressvar{prop:rangehigh} = records(txaq) setcursor(cursor:wait) loop x = 1 to records(txaq) get(txaq,x) ?progressVar{prop:progress} = x-1 clear(ema:record) EMA:TXA = txaq.name Access:EmbedApp.Insert() EmbedApp{prop:sql} = 'select last_insert_id()' next(EmbedApp) ! Add the queue header record access:TextFile.Close() GLO:TextFileName = txaq.name access:TextFile.Open() Access:TextFile.UseFile() set(TextFile) ProcName = '' state = 0 lineNo = 0 clear(procname) clear(lastprocname) clear(lastembedname) clear(currembedname) LOOP next(TextFile) if errorcode() then break. dumptrace = false lineNo += 1 CASE state OF 0 ! search for the start of a module or procedure, or an embed if sub(txt:rec,1,11) = '[PROCEDURE]' clear(vars) state = 10 elsif sub(txt:rec,1,8) = '[MODULE]' clear(vars) procName = '[MODULE]' elsif sub(txt:rec,1,7) = 'EMBED %' embedName = sub(txt:rec,7,len(txt:rec)) state = 30 elsif sub(txt:rec,1,8) = '[SOURCE]' state = 50 end OF 10 ! get procedure name details if sub(txt:rec,1,4) = 'NAME' procName = sub(txt:rec,6,len(txt:rec)) state = 11 end do CheckForMissedEmbed OF 11 if sub(txt:rec,1,8) = 'FROM ABC' procFromABC = sub(txt:rec,10,len(txt:rec)) state = 12 end do CheckForMissedEmbed OF 12 if sub(txt:rec,1,8) = 'CATEGORY' procCategory = sub(txt:rec,11,len(clip(txt:rec))-11) end state = 0 do CheckForMissedEmbed of 30 ! Look for a first embed parameter if sub(txt:rec,1,11) = '[INSTANCES]' state = 41 elsif sub(txt:rec,1,8) = '[SOURCE]' state = 50 end of 41 ! Get first parameter if sub(txt:rec,1,6) = 'WHEN ''' embedParam1 = sub(txt:rec,7,len(clip(txt:rec))-7) WhenLevel = 1 !db.out('whenlevel=' & whenlevel) end state = 42 do CheckForMissedEmbed of 42 ! Look for a second embed parameter if sub(txt:rec,1,11) = '[INSTANCES]' state = 43 elsif sub(txt:rec,1,8) = '[SOURCE]' state = 50 end of 43 ! Get second parameter if sub(txt:rec,1,6) = 'WHEN ''' embedParam2 = sub(txt:rec,7,len(clip(txt:rec))-7) WhenLevel = 2 end state = 44 do CheckForMissedEmbed of 44 ! Look for a third embed parameter if sub(txt:rec,1,11) = '[INSTANCES]' state = 45 elsif sub(txt:rec,1,8) = '[SOURCE]' state = 50 !db.out('found PRIORITY') end of 45 ! Get third parameter if sub(txt:rec,1,6) = 'WHEN ''' embedParam3 = sub(txt:rec,7,len(clip(txt:rec))-7) WhenLevel = 3 !db.out('whenlevel=' & whenlevel) end state = 50 do CheckForMissedEmbed of 50 ! look for the priority if sub(txt:rec,1,8) = 'PRIORITY' embedPriority = sub(txt:rec,10,len(txt:rec)) if lastprocname <> procname ! insert new EmbedProc record clear(EMP:record) EMP:Proc = procname EMP:ProcFromABC = ProcFromABC EMP:ProcCategory = ProcCategory EMP:EmbedAppID = EMA:EmbedAppID Access:EmbedProc.Insert() EmbedProc{prop:sql} = 'select last_insert_id()' next(EmbedProc) lastprocname = procname end ! Add the embed record currEmbedName = clip(embedName) & clip(embedparam1) | & clip(embedparam2) & clip(embedparam3) & embedpriority if currEmbedName <> lastEmbedName lastEmbedName = currEmbedName EMB:EmbedProcID = EMP:EmbedProcID EMB:Embed = EmbedName EMB:Param1 = EmbedParam1 EMB:Param2 = EmbedParam2 EMB:Param3 = EmbedParam3 EMB:Priority = embedpriority access:Embed.Insert() end state = 51 end of 51 state = 60 do CheckForMissedEmbed OF 60 ! capturing embed ! Quit when [END] encountered if sub(txt:rec,1,1) = '[' if sub(txt:rec,1,5) = '[END]' case WhenLevel of 3 WhenLevel = 2 embedParam3 = '' of 2 WhenLevel = 1 embedParam2 = '' of 1 WhenLevel = 0 embedParam1 = '' end state = 0 elsif sub(txt:rec,1,8) = '[SOURCE]' ! look for another embed under this [EMBED] point state = 50 else ! could be we're done state = 0 end elsif sub(txt:rec,1,6) = 'WHEN ''' case WhenLevel of 0 ! get the first param embedParam1 = sub(txt:rec,7,len(clip(txt:rec))-7) WhenLevel = 1 state = 42 of 1 ! get the second param embedParam2 = sub(txt:rec,7,len(clip(txt:rec))-7) WhenLevel = 2 state = 50 of 2 state = 50 end else ! write embed buffer end else do CheckForMissedEmbed END END Access:TextFile.Close() end ?progressVar{prop:progress} = records(txaq) setcursor() |
In the downloadable zip have Embeds-original.zip have a look at the ImportTXAs procedure in Embeds.app for the complete source.
What I had in mind for my new utility was something more along these lines:
My original parser's functionality was overkill for this new app; I really didn't need to build up an elaborate database of applications, procedures, and embed points. I just needed a list of embed points. But obviously I needed all of the parsing capabilities, gnarly though the code might be.
...
And there were other problems. Because my original app was tied to a particular data store (a PostgreSQL database), any re-use of that code would have to know the table definitions. Since Clarion only supports one dictionary per app, any apps that used this procedure would either need to use the dictionary or import the tables from that dictionary.
Class or procedure?
So what's the answer? If it's not a template, and not a multi-purpose generated procedure and not INCLUDEd source, what's left? Procedures and classes, that's what. But not just any procedures or classes. I wanted to write code that had as few dependencies as possible.
...
In fact, a class method is really just a procedure, so using a class already gets you everything a procedure can do. But it also gets you more, because in a class multiple methods can operate on shared data.
The test app
I'm not going to go into all the details of how to create a class; I'll cover that in following articles. For now just keep your eye on the transformation of the embedded code into a class, and don't worry excessively (yet) about exactly how it was done.
The class
The first decision I have to make is how to store the embed data. Originally I used a SQL database, but this now appears to be a liability. My parser should use something more transient, which can be converted to a permanent store or just discarded after use. An in-memory data store, in the form of nested queues, fits the bill:
TxaEmbedLineQueue queue,TYPE
line cstring(1000)
END
TxaEmbedQueue queue,type
embedname string(100)
EmbedLineQ &TxaEmbedLineQueue
END
TxaProcedureQueue queue,TYPE
ProcName string(40)
EmbedQ &TxaEmbedQueue
END
The parser's job will be to populate these queues so that I end up with a TxaProcedureQueue containing one or more records. Each of those procedure records has one or more TxaEmbedQueue records, each of which has one or more TxaEmbedLineQueue records for each line in a given embed point. With that queue structure in hand I can easily update a database or create a text file, as I see fit.
At first blush this looks like an ideal task for a procedure. Pass in the name of the TXA and an empty queue and get back a filled queue: presto! But here's why I think the procedural solution is hardly ever a good solution: there's almost always some new functionality you can add which doesn't fit into the existing procedure.
Think about error handling. You can have the parsing procedure return an errorcode if the parse fails for any reason, but what if you want to get a more detailed error response? What if you want to enable tracing or logging? How would you do that in a single procedure call, without burdening the procedure with some obscenely large number of parameters?
In fact, once I began rewriting my parsing code as a class I ended up with rather a lot of methods and a few properties as well:
...
In recent years I've become a devotee of test-driven development, which embodies the idea that you write your test first and then start in on the code needed to pass the test. I find this provides a lot of clarity to my class design.
So my first question is, what should my test(s) look like?
I'll need a parser object, of course. And this is going to be part of the DevRoadmaps Clarion Library, so it needs to follow the DCL naming conventions. That means that DCL_ is the first part of the name (Clarion doesn't support namespaces in class names, so I fake these by separating name parts with underscores).
The second part of the name? Maybe Clarion, since the parser is a class that relates specifically to Clarion's own functionality. If, for instance, I wrote some code to manipulate solution or project files I'd also begin those classes with DCL_Clarion_. If I'm not expecting a lot of classes then two levels of naming is sufficient, although sometimes I'll go to three.
Provisionally, I'll name my class DCL_Clarion_TXAParser.
The first test
So what should my first test look like? A reasonable place to start seems to be identifying the number of embed points in a TXA, using code like this:
Code Block |
---|
txa DCL_Clarion_TXAParser
code
txa.Parse('test.txa')
AssertThat(txa.GetEmbedCount(),IsEqualTo(6),'Wrong number of embeds found in TXA') |
I'm not creating this code from scratch; instead I'm refactoring a previously created set of classes to use the DCL standards and existing DCL classes. But the original code doesn't have a GetEmbedCount method so I'll have to add that at some point.
The class
The first decision I have to make is how to store the embed data. Originally I used a SQL database, but this now appears to be a liability. My parser should use something more transient, which can be converted to a permanent store or just discarded after use. An in-memory data store, in the form of nested queues, fits the bill:
Code Block |
---|
DCL_Clarion_TXAParser_ProcedureQueue queue,TYPE ProcedureName string(40) EmbedQ |
...
|
...
|
...
|
...
|
...
|
...
&DCL_Clarion_TXAParser_EmbedQueue END DCL_Clarion_TXAParser_EmbedQueue queue,type EmbedName |
...
|
...
|
...
|
...
|
...
string(100) EmbedLineQ &DCL_Clarion_TXAParser_EmbedLineQueue |
...
|
...
|
...
END DCL_Clarion_TXAParser_EmbedLinesQueue |
...
queue,TYPE Line |
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
I won't go into all of these methods in detail - you can have a look at the downloadable source if you're interested. Briefly, there are a few private methods to break up the internal code into reusable blocks (you'd use routines in a procedural implementation), and there are some public methods to get the last error, parse the specified TXA, reset the parser, assign a debugging object, specify the particular queue to use, and write the contents of the queue out to a text file.
Now, show me how you'd implement all that in a procedure!
I now have a Write method that dumps my embeds to a text file, but I could also create another Write method that would dump the embeds out to a database. I'd have to do this by passing in file and field references, and perhaps FileManager references, but it could be done, and it would l ensure the code remained portable between not just apps but also dictionaries.
Automated testing!
Perhaps more interesting to me than simple reuse is that by moving my code into a class I've made it testable. But what does testing really mean, in the Clarion world? Usually it means writing some code in an embed point, running the app, clicking some clicks and watching what happens. That's useful, but it doesn't necessarily tell you what you need to know about the reliability of your core code. And it's tedious.
You can take away some of the tedium with automated testing tools, except that historically Clarion hasn't played well with those tools because Clarion apps use custom Windows controls.
But you still won't be testing your app's core functionality in a repeatable way.
Testing is a complex subject, and there are lots of different ways to test application code, but certainly one of the most useful is unit testing. The core idea is that you reduce your code to the smallest testable units, and then you run tests on a regular basis. Those tests must be automated; they have to run without user intervention (other than to kick off a series of tests, although you might want a test suite to run automatically on a regular basis, or as part of a build).
Unit testing is commonplace in the .NET world, in part because certain features of the CLR (in particular, reflection) make it fairly simple for testing tools to examine assemblies (DLLs), locate classes and methods marked as test code, run those tests and report on the results.
Automated testing is a bit more awkward in a language like Clarion, but still possible, as Figure 2 shows.
Figure 2. Testing the parser class
Testing the parser
Figure 2 is a demostration of two applications, neither of which I've yet discussed. The application that you see running is called (tentatively) CTest, and is a Clarion test runner. CTest's job is to load up a specified DLL, search it for test procedures, run those test procedures and report on the results. It's very loosely patterned on the kinds of unit testing applications readily available to .NET developers. I'll have an article describing the inner workings of CTest next month.
The second app involved in Figure 2 is a DLL APP called TxaParserTest. This APP contains some test procedures created using a special version of the Source procedure template.
Here's the code for a procedure called Test_ParseTXA_OpenFile_ReturnTrue
. First there are some data declarations:
...
cstring(1000)
END |
Since these queues are used by DCL_Clarion_TXAParser I've included them in DCL_Clarion_TXAParser.inc and have given them a prefix of DCL_Clarion_TXAParser_.
The parser's job will be to populate these queues so that I end up with a DCL_Clarion_TXAParser_ProcedureQueue containing one or more records. Each of those procedure records has one or more DCL_Clarion_TXAParser_EmbedQueue records, each of which has one or more DCL_Clarion_TXAParser_EmbedLineQueue records for each line in a given embed point. With that queue structure in hand I can easily update a database or create a text file, as I see fit.
At first blush this looks like an ideal task for a procedure. Pass in the name of the TXA and an empty queue and get back a filled queue: presto! But here's why I think the procedural solution is hardly ever a good solution: there's almost always some new functionality you can add which doesn't fit into the existing procedure.
Think about error handling. You can have the parsing procedure return an errorcode if the parse fails for any reason, but what if you want to get a more detailed error response? What if you want to enable tracing or logging? How would you do that in a single procedure call, without burdening the procedure with some obscenely large number of parameters?
In fact, once I began rewriting my parsing code as a class I ended up with rather a lot of methods and a few properties as well:
Code Block |
---|
DCL_Clarion_TXAParser Class,Type,Module('DCL_Clarion_TXAParser.CLW'),Link('DCL_Clarion_TXAParser.CLW',_DCL_Classes_LinkMode_),Dll(_DCL_Classes_DllMode_) ProcedureQ &DCL_Clarion_TXAParser_ProcedureQueue ProcedureQIsExternal bool Errors &DCL_System_ErrorManager Construct Procedure() Destruct Procedure() AddNewEmbed procedure(string embed,string embedparam1,string embedparam2,string embedparam3,string embedpriority),private AddNewProcedure procedure(string pname),private CheckForMissedEmbed procedure(string s,long lineno,long state),private GetEmbedCount procedure,long GetProcedureCount procedure,long Parse PROCEDURE(string filename),bool,proc RemoveCurrentProcedureFromQueue procedure,private Reset |
...
|
...
PROCEDURE |
...
SetQueue |
...
|
...
|
...
And then there's some code:
parser.SetQueue(q)
txa = '..\Invoice\invoice.txa'
if parser.parse(txa)
tr.Passed = true
ELSE
tr.message = parser.GetLastError()
END
There's some additional code which is automatically generated by the procedure template; I've only shown the embed code I added to create the test. For instance, the tr
object is an instance of TestResultT
, which is a utility class used to report results back to the test runner (CTest), and tr.Passed
defaults to false
. Again, I'll get into the details in a future article.
The entire purpose of this test is simply to verify that the parser's Parse
method execute successfully, indicating the specified TXA was found and something was done with it.
The Test_ParseTXA_GetProcedures_VerifyCount
test verifies that the parser found the correct number of procedures containing embeds. The data is similar to the previous test, so here's just the code::
parser.SetQueue(q)
txa = '..\Invoice\invoice.txa'
if parser.parse(txa)
if records(q) = 13
tr.Passed = true
ELSE
tr.message = 'Expected 13 procedures but found ' |
& records(q)
END
ELSE
tr.message = parser.GetLastError()
END
Test_ParseTXA_GetEmbeds_VerifyCount
is similar but requires some additional data:
expectedEmbedCount long,dim(20)
x long
And the code:
parser.SetQueue(q)
txa = '..\Invoice\invoice.txa'
if parser.parse(txa)
expectedEmbedCount[1] = 1
expectedEmbedCount[2] = 1
expectedEmbedCount[3] = 2
expectedEmbedCount[4] = 4
expectedEmbedCount[5] = 1
expectedEmbedCount[6] = 3
expectedEmbedCount[7] = 4
expectedEmbedCount[8] = 3
expectedEmbedCount[9] = 2
expectedEmbedCount[10] = 1
expectedEmbedCount[11] = 8
expectedEmbedCount[12] = 4
expectedEmbedCount[13] = 2
tr.passed = true
loop x = 1 to records(q)
get(q,x)
if expectedEmbedCount[x] <> records(q.EmbedQ)
tr.Message = 'Test failed on index ' & x |
& ', procname' & q.ProcName & ', count ' |
& records(q.EmbedQ)
tr.passed = false
break
END
END
ELSE
tr.message = parser.GetLastError()
END
Here's another test, this time called Test_ParseTXA_WriteEmbedLog_VerifyExistence
. Here's the code:
txa = '..\Invoice\invoice.txa'
parser.SetQueue(q)
if parser.parse(txa)
if parser.Write('ThisIsATestFileAndCanBeDeleted.txt')
if ~exists('ThisIsATestFileAndCanBeDeleted.txt')
tr.message = 'Output file was not created'
ELSE
tr.passed = true
END
else
tr.message = parser.GetLastError()
END
ELSE
tr.message = parser.GetLastError()
END
This test verifies that the parser was able to write a specific test file.
These tests are by no means exhaustive, but they do illustrate the usefulness of extracting code from embed points. Not only can I reuse that code in other applications, I can test the code and gain confidence that it's doing exactly what it should be doing. This is especially important when it comes to modify the code, either because a bug was found (in which case there should be a new test that confirms the bug fix) or because some new feature is needed.
Any time you change code you run the risk of introducing new bugs; having a comprehensive test suite greatly reduces the likelihood of those new bugs going unnoticed. It's also a great way to verify that software upgrades (whether Clarion, or third party products, or even operating systems) haven't broken your code.
Obviously, this kind of code extraction works best with code that isn't tied to the UI and isn't dependent on a particular database, although in most cases, and with TPS files in particular, you can still run automated tests more easily against a database than you can against a user interface. But that's also my point: the code that really gives you application value is almost always code that doesn't depend on the UI or on a particular physical database. It may depend on a certain data structure, but that structure seldom has to be just a TPS file or only a SQL table or whatever specific implementation you currently use.
Embed code reduction in the embed utility
So what's the end result of refactoring my parser into a class? After exporting the TXA from my ListEmbeds.app utility, I ran that utility against the TXA and got this output:
PROCEDURE: Main
EMBED: %ControlHandling ?TxaName 4000
do SetOutputFilename
EMBED: %ControlHandling 4000
do SetOutputFilename
EMBED: %ProcedureRoutines 500
SetOutputFilename ROUTINE
if clip(txaname) = ''
outputfile = ''
else
OutputFile = clip(TxaName) & '.embeds.txt'
END
display(?outputfile)
EMBED: %DataSection 1300
parser &TxaParserClass
q TxaProcedureQueue
EMBED: %ControlEventHandling ?go Accepted 2500
parser &= new TxaParserClass()
setcursor(cursor:wait)
parser.SetQueue(q)
parser.Reset()
if parser.parse(TxaName)
if not parser.Write(OutputFile)
message('Unable to write output file: ' & parser.getlasterror())
END
ELSE
message('Unable to parse txa: ' & parser.getlasterror())
END
do Viewer4:Initialize
display()
setcursor()
dispose(parser)
There's still a bit of embed code there, but now none of it contains any significant business logic. All the critical bits are inside my parser class.
There is a template that does this already, you know
Even back when I wrote the original parser I was pretty sure there was a template that would extract embed points, and Steve Parker finally pointed it out to me: Bo Schmitz' free BoTpl utility can extract embed points from applications. Bo's excellent template does this by exporting a TXA and parsing the result. Sound familiar?
As useful as Bo's template is (and I highly recommend you get it and use it) I think it also points out the value of putting business logic into classes rather than into templates. Source code can be easily tested, and even debugged with the rudimentary Clarion debugger; templates are much more difficult to test and there is no true template debugger, only logging tools.
No, I'm not done yet
As has often happened to me, in the course of writing my tests I discovered some new features I wanted to implement, and also a bug or two that my initial set of tests hadn't uncovered. I'll explore these issues another time; at some point I'll also detour into an explanation of the CTest test runner app, and then it'll be on to the semi-real-world example of the Invoice app.
The downloadable source is a C7 application containing four APPs and a source code project. You can also load up any of these projects individually.
- CTest - the test runner application
- Embeds - the original embed application from the article series (doesn't extract embed source)
- ListEmbeds - the new app to list embed source
- TxaParser - a hand coded project containing the TXA parser class and supporting code
- TxaParserTest - an app containing unit test procedures for the TXA parser
Figure 3. The solution
As well there are libsrc and template directories containing supporting templates and classes you may want to copy to your libsrc and template directories.
Download the source with DLLs (15 megs)
...
procedure(DCL_Clarion_TXAParser_ProcedureQueue procedureQ)
End |
I won't go into all of these methods in detail - you can have a look at the source if you're interested. Briefly, there are a few private methods to break up the internal code into reusable blocks (you'd use routines in a procedural implementation), and there are some public methods to get the procedure and embed counts, parse the specified TXA, reset the parser, and specify the particular queue to use.
Now, show me how you'd implement all that in a procedure!
Note that use of the SetQueue method. By default the class will use its own internal DCL_Clarion_TXAParser_ProcedureQueue instance, but if I wish I can tell the class to use an external queue so I can more easily process the data, e.g. to write it to a database or a text file.