In other words,  instead of this

<?xml version="1.0"?>
...
<fullquery name="db_api_acceptance_test_select_asdf_sysdate_from_food_2">
      <querytext>
      select asdf, sysdate as datestr from footest where asdf > :asdf
      </querytext>
</fullquery>

which is not valid XML (and thus chokes tDOM),  why not use this:

<?xml version="1.0"?>
...
<fullquery name="db_api_acceptance_test_select_asdf_sysdate_from_food_2">
      <querytext><![CDATA[
      select asdf, sysdate as datestr from footest where asdf > :asdf
        ]]></querytext>
</fullquery>


I'm doing a regsub find-replace to make them into parser-friendly xql,  but I'm wondering if fixing them at the source is an option.  Non-standard xml seems to defeat the purpose 

I asked Don about this before when I wrote some validation utilities for the xql files. I think his view was that it was a little annoying to write correct xml (manifestly true) and that turning all the extant files into valid xml was not a high priority.

I did validate all the files in 4.6 at one point so a part from the quoting problem they should now all be clean (and QD does an ns_quotehtml on the query text before it feeds it to ns_xml so it is actually parsed with a standard xml parser rather than something hand rolled).

I think it's easy enough to clean up with a simple script, maybe it's time to do it. I would also like to do some stuff to cache the parse across restarts since I would like the server restart to be quicker and as it stands the qd parse is an appreciable fraction of the startup time.

Could you create a new mega-file with all the queries, or maybe just a tcl file that runs a series of commands, avoiding parsing completely? Seems like the tcl file could check mtimes first and only parse new/changed stuff.

Yet another issue that we need to tackle soon is to drop libxml and switch to Zoran's tDom AOLserver module.  It was supposed to be "released any day now" late last summer so one would hope it's actually available now? 

Don, tDOM 0.7.5 was released 2002-11-27 and is available from http://www.tdom.org/. It works great with AOLServer!

Let's switch over to tDOM. And now that we are on the subject of XQL files, could we please include an XQL DOCTYPE at the top of each XQL file? I can provide a DTD for the type. With a DOCTYPE in place XQL files are a snap to create/modify by hand when using Emacs.

/Bart

I've written a tiny compatibility layer mapping ns_xml commands to tdom commands as part of my portable.nsd effort.  It's not quite drop-in,  but close.

• Mysteriously broken code which would result from errors in the parser or quirks in the particular source tcl.
• even more difficulty in merging improvements from particular sites back into the code base.
If it were a cleaner language or if we had a full parser I would not worry quite so much but regardless of how good the parser is, the later problem will remain.

I don't see a lot of evidence that the QD overhead is anything to be that concerned about relative to going to the DB in the first place.

I think there is probably a better argument for precompiling adp files rather than doing them on the fly, although there it is much clearer that what you are doing is compiling and no one would really be tempted to discard the adp's and work directly with the resulting code.

People who are developing multi-DB applications will want use query files.

The QD overhead, as Jeff suggests, appears to be very low when compared to the expense of executing a query.  There's really nothing to be gained here performance-wise.

It does require some memory ... on OpenACS.org about 3MB IIRC (we looked at this when trying to figure out why our site was eating up so much memory).  On a modern machine that's not a large number.  And if we follow through and pull queries out of the Tcl files belonging to standard OpenACS project supported packages of course the compiled Tcl bytecode will shrink by a roughly comparable amount so the overall overhead is very low.

The single issue which I think all would agree is important is the start-up issue, as start-up time can be fairly slow on sites that use a large number of packages.  It's not just query files, though - we're compiling every */tcl/*-[init/procs]*.tcl file on startup, too.  It's not clear how much can be done about this.

I really don't see much value in a tool that folds .xql files back into .tcl files, to be honest.  It's not something I'd ever use.