I like the idea, Brent, since I have gotten to liking the Debian approach to upgrading your system. Unfortunately, there is good news and bad news on that front. The good news is that at least one ArsDigitan is working on templates for the ACS, I think in much the way that you propose. It is described as:

The dynamic publishing system is a substrate for team development of web applications. It maintains a strict separation between data processing and presentation, allowing developers and designers to work together more effectively. The system handles both content pages as well as most forms.

The bad news is that at least one of the other ArsDigitans (Jin) has said that separating out Oracle-specific SQL is "a particularly pernicious idea" (his words, not mine - see http://photo.net/bboard/q-and-a-fetch-msg.tcl?msg_id=000WhO for more - look for "pernicious" . He says that that kind of separation (not quite what you propose, but still something that would be handy for us ACS/pg types) slows you down in producing new sites, and that being able to swap new RDBMSs in and out isn't particularly useful.

That said, I'd love to see the kind of abstraction that you propose done. I confess to finding it a daunting prospect, unlike some folks here who seem to eat, drink, and breathe ACS (I'm working on it, I promise!). I also have some trouble seeing how we could manage to do it on our own and manage to keep up with ArsDigita's paid army of programmers. Of course, on the gripping hand, we do seem to be growing in numbers of downloads pretty quickly!

As far as Jin's comments regarding supporting more than one database, well, do keep in mind that aD has a very narrow mission, to become the  dominant developer of complex database-driven web sites.  Most of their potential customers use or will be using Oracle.  aD is tight with Oracle.  Supporting multiple databases would simply slow them down, without tangibly increasing their business and potentially putting their relationship with Oracle at risk.  I suspect the latter reason's the most important, to be honest.  Philip loves Oracle.  Oracle bought an earlier company of his.  Several months ago, when a curious individual e-mailed him about the desirability of making the ACS available on a low-cost or free database like Postgres, Philip lambasted him in his response, telling him that only "losers" would consider using anything other than Oracle.  No, it wasn't me, but I did get to see the e-mail exchange...

I think aD is starting to wake up to the fact that a Postgres-based ACS might turn out to be very popular, indeed.  While I don't envision  ACS/PG taking over huge sites built on load-balancing server farms (the fact is that PG has zero support for this level of scalability), I do envision the ACS/PG being used to implement a large number of more modest sites.  You could do something like scorecard.org using ACS/PG, for instance...

The bottom line is that any such abstraction of queries would have to come from us, would probably not be folded back into the mainstream ACS, wouldn't be maintained for new modules by aD anyway (we'd be porting to this abstraction), and the needs of ACS/PG users may not track mainstream ACS forever anyway.  We have no way, at the moment, to port over Java embedded in Oracle except to rewrite it or find someone with the time to embed the JVM into Postgres ala the Tcl and Perl embedded languages - any takers?  Webmail's being implemented by aD in this way, tying them even more closely to Oracle (and it's being  implemented by Jin).

Jin's comments should also be considered in light of the fact that aD is apparently working on an Apache/mod_perl version (or at least is considering it).  The use of AOLserver shuts them out of a fair number of potential customer sites, since Apache and Perl are both objects of  religious significance to a fairly large slice of the Unix world.

This will slow aD down.  Maintaining AOLserver/Tcl and Apache/mod_perl  versions will be a far greater maintenance headache than trying to support a couple different dialects of SQL.  More evidence in my mind that it is the relationship with Oracle, not technical considerations, that causes aD to refuse to help make the toolkit more portable in regard to the underlying RDBMS.

Even Phillip Greenspun himself has spoken of porting the ACS to *gag* WindowsNT and *retch* Microsoft SQL-server.  And these would certainly be far more involved and labor-intensive than (as Don points out) modifications for different SQL dialects.

So one question that immediately springs to mind, is if Don's conjecture is true (that a database-agnostic system would not be of interest to aD, and they would be unlikely to fold such changes back into the source tree), then where does that leave us?  When Ben suggested that ACS/pg might someday just be a feature-equivalent port of the ACS, rather than an architecturally-identical port I was somewhat shocked, but after learning more I wonder if this isn't perhaps the best option.

Perhaps I am just naive, but I really think it should be possible to migrate a website from one version of the ACS to another without having to hand-code every change.  For example, at my current employer when we make a large change to a database construct that changes the way existing data should be stored, we have to write a "Rel" program (release program) that will walk through the database and update these values.  These kinds of tasks should be possible without having to redo it for each and every site.  However, I might not grasp the whole of the ACS's architecture (in fact I am sure I do not) and could be missing a key point.

Also, if the Tcl really is just plumbing, then I sure hope that Arsdigita will start abstracting out the *real* ACS from this "plumbing" ... such is not even close to the case right now....

On the other hand, muddled code is pretty common in fast-growing projects where time is of the essence and deadlines tight.  The ACS isn't the only software project to suffer from such constraints.

As far as whether or not an object-oriented approach to separation of queries, programs, and HTML makes sense, I've not really thought about  it.  I do believe the object-oriented paradigm would be useful for designing a reasonable model for module add-ins, and a natural way to think about providing easy-to-configure ways to define permissions, etc, for various modules and parts thereof.

As far as the templating problem goes, the problem lies in mapping the object-oriented paradigm to the hyperlinked, page-oriented reality of the web.  I've not looked at Zope but it's not hard to imagine that such a scheme could get complicated in a hurry.

Ben -- Tcl is great: it works well for its current purpose, and it's extraordinarily easy to embed. I agree that requiring another language is a bad idea -- but I think that "OO Tcl" would end up being very much like C++, a hodgepodged mishmash of ugly code that only recently (after how many years?) has compilers that fully support it ;).

But if Python allows embedded Tcl (which it does, with very little effort) then there is a good reason to build an OO framework in it and have Tcl code working within that framework, instead of trying to build the framework with a tool that's ill suited to it (Tcl) or with an extension to that tool ([incr Tcl]) that's grafted on. Moreover, a well designed templating system shouldn't require anything but very serious infrastructure-style hacking -- and people interested in doing such usually don't mind learning new languages.

As an end to these language wars: I'm working on a few projects with PyWX, and I'll post my results here when I'm solidly on my way. It may be that working with Python isn't a great boon and merely complicates matters (which, in some cases, it does); whether or not these things can be overcome is a matter for the developers of PyWX. It's "our job" to bring it to you, not your job to come to us ;). But if you're going to argue that Turing completeness is enough for a language to possess, let's talk APL, PL/SQL, Fortran, x86 assembly language...

Cheers, --titus

Check my comment again. I recommended *both* against OO and against Python for the ACS. We have 1 million lines of Tcl code or so, and plenty of problems already. I do not believe that a "more extensible language" or that OO is going to solve our problems. Our problems need plain and simple software engineering work, and in that respect I like the KISS principle. Adding a new language, not matter how amazing and beautiful, will only decrease the number of potential users and increase the complexity.

I recommend you take a closer look at Tcl. It's a lot more flexible than you seem to imply, and it might be faster for you to learn more about Tcl (http://photo.net/tcl) than for you to fine tune your performance and thread-safety in Python...

Ben -- I absolutely will spend more time learning Tcl.  However, increasing the performance and thread safety of the Python engine is (to me) a more interesting problem, and one that I started independently of the ACS.  This work will continue.

However, I recognize that the ACS is the most powerful Open Source toolset available for developing web sites.  Perhaps Zope has some useful features, perhaps not.  I do know that the only "large scale" websites that I know of using it have only a fraction of the functionality of even the ACS/pg sites (which don't fully implement the ACS stuff yet).

I am reminded of the "UNIX way", which has always been to use lots of small tools that do a particular job very well.  Perhaps Tcl is that tool for the ACS.  Perhaps Python might be a very useful tool as well.  Perhaps the ACS would be most useful if we used both.  I can't say.  I do know that your average UNIX Sysadmin must know lots of little tools.  No one says "I'm only going to use shell scripting.  No AWK or SED for me".  It would be laughable.  So I'm not sure it would be as crazy for a website architect or administrator to need to know Python as well as Tcl.

Furthermore, we can draw a distinction between some sort of mythical Object Oriented framework written in Python that uses Tcl, which organizes and coordinates the various Tcl/adp pages, and the Tcl/ADP themselves.  We don't complain that you have to know C to modify the AOLserver engine, do we?  Why complain if another loadable AOLserver module is written in Python?

At any rate, I just want everyone to know that we don't want to cause a language war and that was never our intent in starting PyWX.  I would prefer to talk about what we *can* do to make the ACS more modular.  Since this thread is getting quite long, maybe I'll post under a new topic.

Why am I here? Without spiralling too far from the point of this thread: 1) Zope is still relatively immature in terms of prepackaged solutions (like commerce, sessions and user profiles), 2) though benchmarks show relatively good performance, and we've had fine performance so far, I do wonder about scalability, and 3) I've always got an eye out for other tools. One comment about scalability: their going to release their replication engine (ZEO) as Open Source in May. Getting bogged down? Throw another machine at it! Granted, this in and of itself does not solve issues with python's thread safety, or the fact that the TCL interpreter is likely to be faster than python's.

The existing templating capability in Zope is pretty powerful. There has been discussion of new extensions lately to try to make it play better with page editing tools like Dreamweaver.

One powerful feature of Zope is called "acquisition". Everything in Zope is an object, stored in the ZODB. (Currently, ZODB is a single file that gets appended to with each change. It is possible to store the ZODB in a relational database, but no one has done this yet.) Every object in Zope can be traversed to directly via a URL, and methods can be called on that object by simply adding the method to the end of the URL. The ZODB itself is hierarchical. Acquisition lets objects that are farther down in the hierarchy access objects that belong to its parents.

What does all of this mean, practically? Most documents in Zope do a <dtml-var standard_html_header>, which will call the method standard_html_header. You can define one of these at the top of your site, and everything underneath will access it. If you feel like having a different design in a different part of your site, just define a new standard_html_header there.

Another thing that I really like about Zope that has content management implications is that objects can contain other objects. So, the catalog item objects on my site can contain any number of images, and I can easily iterate over the images to display them.

Though the ZODB is hierarchical, Zope has an indexing system called ZCatalog. This allows you to index data on whatever properties you choose and do (relatively simplistic) searches on that data. This is a way that you can get at data regardless of where it lives in the hierachy. On Zope.org, each member can author how-tos in their own space. After review, their how-to becomes visible in the main documentation area of the site (and to site-wide searches, etc).

DTML has its shortfalls, but it also has pluses... like <dtml-in>, which can iterate over objects easily. It can iterate over subobjects, SQL results from an RDB, ZCatalog search results, lists from python, etc.

I'm not trying to do Zope advocacy here or anything. I just thought I would toss out a few bits about how Zope does its thing. I do agree that Zope has a bit of a learning curve... if there's anything you're curious about, feel free to ask.

Kevin

I'd also like to note that PyWX has gotten a nibble of interest from several ArsDigita people. The main objection I've heard is that they're too busy writing sites to worry about the language, which I suspect is going to be the dominant objection.

cheers,
--titus

PS the main limitation, for me, is reusable & extensible code.  I find myself writing the same damn functions over and over again, with minor variations... ugh.  It's like I'm writing in C again...

I don't think we've done anything even close to the "right thing" so far, with Tcl as the programming language and with almost no complex work taken on by the ACS core (i.e. the developer interface to the ACS is not exactly simple). So, I think sprinkling in a bit of the "right thing" won't hurt too much. It's not like we're talking about using Lisp :) We just want to spend some amount of time thinking about how best to go about this which, granted, is a far cry from the barely-designed ACS 3.x ....

I think what you're looking for here is an application generator with enough juice to generate the ACS from a specification. That way, you could port the ACS to any platform by altering the generator code. This would also have the benefit of standardizing the database access, page display etc. methodologies that have been cobbled together over time in the ACS.

I've gone over the work that Karl has done on the templating system for the ACS. It's very good. The purpose of that system is to allow a programmer to write a text document using an XML formatted schema. That document would describe the application in question. Looping, if and other constructs are then made available to the web designer that allow the web designer to fold the input and output fields of the application into the page in any way he or she wishes.

I looked at that and thought "Hmmmmm, if this XML structure was replicated as database tables, then we could throw some reusability in here." (Well, sort of, I'd already started on my design when I found Karl's stuff)

I believe that Karl and Philip have gone back and forth on whether the specification should be in the database or in XML. The XML approach is good because you can represent complex structures that are hard to represent in a database. The database approach is good because you can edit it online and have background processes that check your work.

I've got some sort of a database application generator running right now. It generates the pages but they aren't intelligently linked to each other. I'll have added the navigation components fairly soon. The basic idea behind it is that there aren't that many ways in which data is displayed. So, I write a display generator for each method. So far, I've written generators for List, View, Edit, and Add. Once I get those up and running, I think I'll stop and try to generate either Wimpy Point or the Ticket Tracker. (The ticket tracker is probably the candidate because Wimpy Point needs a linked list viewer and an archiver.)

The generator expects to find the description information it needs in the database when called. If the description information is there, it generates a page of code. If you want to add another field to the database, you describe and generate it. If you want to add a method to the application, you describe and generate. Anyway, that's the theory.