Bart,
It's a bit tough to give much in terms of pros and contras od nXML vs. PSGML -- when editing large documents I do prefer the former, but largely due to periffery, e.g. Norman Walsh's unichars/xmlunicode that allows to just type EM dash, instead of putting in —. I don't suppose on-the-fly parsing and validation is very critical for XQLs.
On the flip side, named entities are not supported in nXML the same way as you may be used to with PSGML, i.e. if you declare ENTITY OACS somewhere in the topmost file of your DocBook project (cross referenced in PSGML way via parent-doc), nXML won't understand it, and hence will claim your doc to be invalid.
It will probably be a bit tough, and may in fact require some rework in Templating and possibly other core parts of OpenACS to make ADPs into a valid XML. In particular, I am not sure how to make @vars@ accpetible as attributes in <if> tags -- I don't think XML allows '@' in the name of an attribute.
The reason for my question was more related to a (selfish) fact that I've ripped PSGML out from my .emacs 3-4 months ago, and am not very much inclined putting it back... I'll have a look at xhtml.rnc of nXML to see if it can be hammered into validating ADPs.
Lastly, I think that James Clark has a tool that does DTD->RNG transform, hence having valid ADP and XQL DTD, it may be fairly trivial to create a basic RNG schema... Yeah, here: http://www.thaiopensource.com/relaxng/trang.html