It seems to me that the top incentive for volunteer coders to pick an
environment/toolkit/language is its groove.  I suspect I'm speaking
for a lot of other Open/ACSers when I say that I started using the ACS
toolkit because I got into the groove very easily.  I could code
working database-backed pages, and they worked, and when I thought of
an improvement, very often I could do it and see it in a few minutes
or less.  And backing this up, I could see that the groove wasn't a
dead end - that this effortless code had a fair amount of stability,
reusability, and other good stuff.  That I wasn't just making cheap
progress that would come back to haunt me, but that I was using a
well-engineered system.  (Or at least, a system with a lot of
promise.)  It was a long groove.  And that was a rare combination in
1998, and it's still rare or there wouldn't still be people committed
to this toolkit.

What's made Open/ACS coding a bit of a love-hate thing is that we have
a rough balance between things that make coding a groove thing - quick
and easy success and feedback - and things that make it good long-term
engineering - reusability, encapsulation, and the other buzzwords.
It's always been a rough balance, and perhaps the early success was
sheer luck.  I don't know if there's a consistent long-term trend, but
I'm concerned that development by and for the high-end developers -
the very people who do the heavy lifting in the community - may be
making the groove harder to find for new entrants.  Maybe we can dig
up the aD forums and try to do some statistical analysis over the last
five or six years of the frequency and intensity of newbie
frustration vs joyful posts.

I'm not sure I have any useful
prescriptions for this beyond what we're already doing with
documentation and the tutorial - trying to make a clearer trail to the
groove.  A few people have made some great contributions to the
tutorial, such as Simon's automated testing intro and Jade's many
comments.  I'd love to see more pieces of tutorial get contributed.
Maybe people could pick a few of their own packages, that are in
production in the real world, and choose a few pages around which to
write tutorials for interesting techniques.  Or people could pick some
core modules and write tutorials explaining bits of them.  In terms of
API improvements, I'd like to see it even easier to build the basic
unit of a database-backed website: a sortable, filterable list of
records combined with one page per record add/edit screen.  I would
like to back away from the recommendation of xql for everything, and
yank it from the tutorial in favor of xql only for pages that need
it.  Making things ACS objects is more useful than even before, since
now you get not only permissions but also comments and notifications
almost for free; someday soon maybe categories; can we eliminate all of
that ugly, hard-to-debug, lots-of-work-with-little-positive-feedback,
groove-killing PLSQL code required to Objectify records?  Maybe code
generation?

Anyway.  food for thought, I hope.  Which parts of coding OpenACS
packages still provide groove?  Which really don't?

Joel, I agree 100% code needs to be easy to write. Actually why write it at all? For simple applications there is no need. Several efforts have been made toward this. There is some code in the core associated with groups. The CR has a different take on the same problem. Both auto generate pl code and data model.

As usual I disagree with both of these approaches for one major reason: you never write down the data model, so it can't be very useful. You never write down your pl code, so it can't be very easily debugged, and you can't add anything special.

So when I started using ACS4, I wrote a package to help me write Oracle packages. When I started using OpenACS and writing in pg, I needed to port code from Oracle. So I wrote a few procedures to make queries in pg as simple as in Oracle, that is the object.new type queries. The procedures are actually used in the cronjob package, with examples in the /www pages xql files.

Then I combined the two ideas into query-writer. It was a difficult to use package with no documentation, but it did exactly what I wanted it to do: write pl code and handle inserting and updating data.

Recently I upgraded the package to be more user friendly. Now all that is required to create a pg package is to write the data model. There is a wizard where you supply the name of the table, the primary key and the name of the 'object' you want this to be. The wizard rummages through the database and loads the query-writer tables with all the necessary data.

Another improvement is the almost complete elimination of pl code. From the information already provided, the pl code which creates the object type is written, plus an object_p function is created, and a drop script is written as well. All object create,update,delete pl is eliminated. I achieved this by just including tcl apis for acs_object__new, etc. This is very nice as when you realize you have forgotten a table attribute, you just add it without having to drop and resource the pl functions. None of this precludes you adding pl functions as before. But if additional code needs to run that can be based on the attributes inserted into the object, you can use a table trigger to get the same effect.

The tcl api for the new object still isn't written down, and I need to address this issue so that query-writer could be used just to generate apis.

On the downside I have never ported to Oracle. It does write the pl code for an oracle package, but the datamodel is for pg. Also there are 8 little tcl procs that need to be written to handle the generation of insert/update/delete in Oracle. This is probably a 3-5 day effort overall, but the package isn't feature complete, why port it?

Query-writer is undergoing some major changes at the moment. qw_groups are being tied to rel_segments. A qw_group is more like a role, an idea that is best supported by a rel_segment. Currently there are two groups: admin and the default group. Qw_groups encompass a set of objects+attributes+values+operations. If your application need more than two roles, you can now create a rel_segment and add the user to the rel_segment to give them the role. Another major change is support for file uploads and sub-attributes. If you object has an associated file, you can write a handler for the upload (examples exist). The handler could write the file to disk, or stuff it into an attribute field. Sub attributes were needed for dates. But sub attributes also required multi-attribute filters, to both combine and remove attributes. Until now query-write didn't support filters on more than a single attribute. Now there is a place to run filters or other code after all attributes have passed validation. Additional call back locations are being created as well.

Overall the query-writer form processing page acts like a generalized ad_page_contract. It enforces the same filters on objects and attributes whenever these are created or updated. (But note that query-writer also enforces a 'role contract')

Tom, I think your query-writer is a very interesting project. I don't have time to look at it right now, but when it is in a state that you think even beginners could use, please put it in /contrib.

I'd like to take a look at it at some point.

I think a tutorial showing what it can do would convince a lot more people to use it.

Maybe we can put it on a Wiki once we get that set up. :)

Jade, if query-writer becomes feature complete and I port it to Oracle, that might be a good time to consider adding it to /contrib. Until then I think it mainly serves as an example that achieving a rapid development environment is possible, at least technically.

I've spent a lot of time recently looking at other development frameworks, mostly related to CMS products. Technically OpenACS is in the upper tier of available products. Philosophically OpenACS is nowhere. There is no coherient vision on the OpenACS way of doing things. For instance a lot of Java frameworks stress Model2 or MVC ( Model-View-Controller). OpenACS does a better job of this, I think, than what Java applications are achieving with JSPs or even taglibs. But we are not talking about this, or how it is being achieved, or not. It simply isn't on our language list.

Additionally, although I'm not an xml lover by any means. This is still an important technology. It would be a nice, useful feature if package ui created xml which could be transformed via xslt into the various formats needed. This is something else that doesn't seem to be on the radar screen.

Related to this, is the lack of discussion on how groups of developers, designers, and managers can work togeather to build and maintain a site. It is actually pretty easy to throw togeather an app on OpenACS, however maintaining it, or adapting it is not well covered. Since packages will sit around and be reused and updated for years, you would think this would be a huge concern to the OpenACS community. It shouldn't be a mystery why folks constantly create packages that do almost the same thing as other existing packages. You simply cannot maintain code when it is so hard to figure out why it was written the way it was. Again, this is mostly due to a lack of good, well understood philosophies for development with OpenACS.

LMAO !!

I just read the latest column of "I, Cringely", which talks about the different view of M$ and Linus concerning the coding abilities of a company vs. many distributed developers. At the peak of it's time AD had I'd say around 100 coders working on ACS TCL in client projects. They did not care that much anymore about what the outside world was doing, but drove their truck at a speed defined by marketing and project management (aka. business guys). Though a momentum like this is worthwhile it bears the risk of releasing code when it is not ready to be released. Which, in effect, alienates other developers because of the crappy maturity of the code, regardless how easy it is to get going with it. So I do not think that the existance of aD contributed to the groove of ACS. The existance of Philip, the marketing machinery and the fun environment of people with a huge ego, but willingness to help (as part of their ego) did attract some people to ACS, more than the existance of AD.

Now, looking at this community, I see we are releasing well tested code that has put a lot of thinking behind it. What we lack though, is to release the knowledge, how to use the code, as the methods got considerably more sophisticated. Furthermore, what happened to the concept of Bootcamps. I know, there is no AD anymore, that could run a bootcamp on it's own (well, okay, we do it occasionally, but that does not really count, as we did not open them up to the public so far), but why did this stop. Furthermore, how many universities are using dotLRN? How many are teaching how to use it?

Getting fresh people into the toolkit through a university course or a bootcamp will give you a lot of feedback, how to make it easier to develop with OpenACS. Having a teaching forum for exchange between bootcampers and university students across bootcamps or courses, would be a good place to collect all this feedback, which in return could make it back into the documentation effort and maybe even into improvements for the toolkit.

And would it be too much to ask for developers in the community, to commit one week of their time to teach a bootcamp? Or give a class at a university next to them.

One of the strengthes as Walter pointed out, is dotLRN. But not necessarily only because of the (monetary) lifeforce (though it looks like that most developement nowadays is dotLRN driven, at least if you look at hours spend on development), but because of it's potential to attract new developers and become a standard in the world. I found it very intersting that the language with the most translations in the latest CVS checkout is Arabic. I doubt Blackboard or WebCT would have an easy sell in the Arabic markets nowadays.

Oh yeah, for historical context...

aD didn't just suck ass at the end.  It had problems from the get go, but the bigger it got, the more it sucked.

I learned much from aD's way and still think that something like it could work now, if only founding principles would include rejecting external investments to the company --investers could almost as easily invest in a project that uses the company resources instead.

Part of the aD way, included having two sw developers per package, which helped increase documenting in code etc.  Are there two package keepers for each Openacs package now? If not, maybe efforts should be made to increase participation?

Dirk,

I am not sure I am against a Tcl proc that returns the results of a query. The main reason is that I know where to go to find the query.

Besides that let's discuss the reasons for OpenACS 4. What problems was it meant to solve. After that, let's see if those are still the goals of OpenACS. Then we can decide where it is sucessful and where we can make improvements.

Hi,

I am not totally against Dirk suggestion to on not to use tcl proc to do database work.  But I took the other route of using tcl proc to do database work.  Upon getting some suggestion and indication that is the path to go.  I decided to go that way.  Here are the pros and cons I have encountered:

- makes it life harder for the one making the tcl proc.  You have to think about what to make and how to present it.
+ makes life a lot easier for those who will use.  they don't have to know the lower layers.
+ make one object manipulation a lot easier. that is add, edit and delete.
- makes listing of objects harder.  I have actually decided that the listing of objects be only used for preliminary development or basic behaviour.  the end developer should just use the query as reference and create his own select based from the reference query.
+ optimizing or bug fixing becomes easier, as once the tcl proc is fixed.  code that uses it get fixed too.
- as more people/code use it.  you will have to more and more worry about compatibility.
- added burden for you to review your code.  which tcl proc to merge, split, bad presentation, etc.

Anyway I hope the following info may help both group people looking at moving db work to tcl AND moving db work away from tcl.

I think given that there are many of us, its likely we will have different opinions.  The OCT should be able to give direction what is best of OpenACS in general.

I am not yet completely sold to the idea of moving db work to tcl proc.  But so far I think for my current use despite the negatives, I value the positives more from what I got from it.

There are conflicting goals. Allowing easier access for experienced programmers and a shallower learning curve.

I definitely want to be able to resuse queries. The reason I have moved them to a tcl proc is I find I am using the same query in every page of a package.

I don't see how having a query in a tcl proc makes maintenance harder at all. Just different. The concept is that someone who wants to modify a package could make changes in the tcl/adp pages without diving too deeply into the database.

Are there examples of code that is difficult to maintain because of queries in tcl procedures?

I think most dml should have a tcl wrapper function (permission::grant for example) and for display queries the reuse should be via includable .tcl/.adp pairs. Of course, no rigid rules like that work in all circumstances...

A big advantage with having dml in tcl procs is that it is much simpler to write unit tests and in most cases the dml is for a single object anyway. I would agree that for things where you are operating on a collection of objects you probably will want to do it directly in sql rather than create a tcl wrapper function (although even there I think an api that takes a sql fragment or list would be useful and more easily tested).

Tilmann,

Excellent points.

Application coding should have different priorities than a service package which function is to provide an API.

We seem to be having this same discussion among the same folks :)

So I think we all agree that efficiency is of top priority as well as maintainability of code.

Right  now there is no easy way to resue queries except by encapsulating them in a Tcl procedure.

We are in agreement that often you just need the name of one value out of the database, and an additonal layer of abstraction gets in the way.

Does that sound right?

This might change if more packages used more central services to create and manipulate acs_objects. One example is the content repository. Getting a list of items is not the best in a tcl procedure, but for common manipulation of one object at a time it seems to work.

The reasons why I personally like the idea of Tcl procs for simple queries is the same as why I like reusability in general, i.e., less coding and increased maintainability. I fail to see how maintaining one Tcl proc and its associated common .xql file (or separate PG and Oracle *.xql files) can be more work than maintaining any number of clones of the same *.xql file(s) in all places they occur within a package (say) ...

Having said that, I am certainly not promoting the use of a Tcl API which calls something like "forum::get" in a foreach loop - those fields should, of course, be retreived from a separate query. OTOH, just because the previous is true doesn't mean "forum::get" has no use ... it might be useful in other places.


Also, we have a mechanism called the query dispatcher for a reason; the ability to support multiple dbs. Let's not go out and tell people that it's fine to skip this mechanism just because it's possible to do without it, or at least bypass it (by writing the queries inline). Let's instead encourage people to uniformly adhere to the standards we have once set out to follow!

As has been said elsewhere; we need fewer ways to achieve common tasks in this toolkit, not more ... What we need more of, though, is coding *standards* - not something vague like "best practices"!! This is how we lower the learning curve, by telling people explicitly how things are done.

What's most convenient/easy for an experienced OACS developer ought to also be easiest for a newbie, no?

/Ola

Neither Til nor I want to copy the same (or similar) SQL query allover the place. As Til notes: OpenACS need to know SQL. They need to know it better than TCL in fact.

If all SQL queries were moved out into the XQL files we would have something like a database "persistence layer". Having those queries in their own respective files would make us look at similarities and probably reduce the number of different queries and make the toolkit and the programmer more scalable.

So can the query dispatcher pick a query from a randomly named file or not?

(Coding SQL is more groovy than coding TCL btw :)

I agree that DML should have a wrapper to support unit testing. But we don't need a wrapper proc on both TCL and SQL...and there is a unit testing package for Oracle SQL (by Steven Feuerstein - utlplsql or sth similar). TCL procs are easier to debug, PL/SQL procs are easier for migrations. Oh whoever thought that PL/SQL is easier to port to other databases - I guess he/they were wrong ...

We are much closer to being able to support rapid development than many believe. While Tom's query writer is one way to go, we can already generate content almost automatically from PROPERLY DECLARED content types (not general object types).

Here's a code snippet ...

        # Form creation hack!

        ad_form -name new-user -export {
            {parent_id $some_parent_id}
            {name $some_name}
            {other_stuff $some_other_stuff}
            {content_type phb_person}
            {content_method no_content}
        }

        # create the photobook user record.
        set id [content::new_item new-user]
        cr::item::set_latest_revision_live -item_id $id

Boom! A new content item with its latest revision set live ... now from this hack to an extension to ad_form that properly allows one to integrate properly defined CR types is not a large step IMO.

As far as xql files are concerned...I'm very, very loath to give them up. In fact I'd love to (in the far future) only pass SQL directly in the db_* API if one uses an explicit -sql switch - only for those rare cases where we need to generate dynamic SQL inline. Of course it's fine for one-off packages or packages that will only support a single RDBMS to avoid .xql packages - but this doesn't define packages in the toolkit itself other than some of the probe-the-db-schema type package work that by its nature is 100% db dependent.

But creating objects shouldn't require writing PL/[pg]SQL and already doesn't, actually - there's a primitive object creator for properly defined objects and as I mention above CR items have much more capability (if they're defined properly).

Now currently you have to "properly define" such things in PL/[pl]SQL, notably the tedious defining of attributes for CR items, but this is yet one more thing that can be done via a TCL API in a declarative way simplifying things ...

Regarding xql files I meant to say that I don't use them in the beginning when prototyping a package or when I'm working on a specific package for a client that will always use one db only and is not intended to be redistributed, that's all. The ability to write inline queries makes developing a little quicker but of course for general packages xql files should be used. Btw, making inline queries explicit with an -sql switch sounds great and would eliminate lots of confusion for newcomers too.

Ola, forum::get has the other drawback of being slightly inefficient by selecting more fields than needed, propably it doesn't matter in this case at all, but it's symptomatic for the dangers of redirection via tcl apis for queries. I would prefer using something like "set forum_name [db_string get_forum_name {}]" instead, possibly with a reused xql file if that query is used more often.

From your example it looks like the use of the query writer as you suggest also removes the usage of the templated form builder.  This is, I do believe, a very bad idea.  Having worked with a large site built with custom built-by-hand forms which saw the .adp files shrink by a factor of (I kid you not) roughly 500 times when moved to the use of a single form builder template shared among all forms, I will never go back to writing my own input tags again.  Ditto for giving up the kind of validation made easy by the form builder/ad_form.

What's wrong with using object metadata to drive things?

As far as this comment goes:

"Differences from the cr include the fact that you write down your data model: a real table with real constraints."

I must say I'm at a loss to understand it.  When one customizes a content datatype one writes a real table or allows the attribute creation code to create the table for you, but in both cases your custom type has a "real table with real constraints" resulting.

As far as comments above regarding the fact that we have too many ways to do things with little documentation as to which are best practices and which are deprecated/frowned upon, this is absolutely true and something we should strive to fix.  All forms in the system should be built with the form builder and preferably ad_form, for instance, which in almost all cases greatly simplifies their creation and processing and makes easy the task of providing forms within a site with a uniform look and feel.  On and on with item after item we should standardize our approach.

But of course lots of people don't like the form builder, are uninterested in using the content repository, are uninterested in exploring the use of object metadata to automate the boring stuff, etc etc.

We have people who want to build square, pentagonal and hexagonal wheels as well as reinvent the traditional round wheels to be found in the toolkit.

So I'm not optimistic about our ability to settle on a handful of best practices used throughout the toolkit.

Query-writer can be used with the form builder. You just fill an array with the attributes/values and call the query writer proc.

The query-writer url api has built in per attribute validation. I'm currently adding multi-attribute validation. It uses the same filters as ad_page_contract. It also automatically does a length check on char/varchar attributes, although now I think there is an ad_page_contract filter that does the same thing.

Using the query-writer url api ensures that the same filters are applied every time an object is created or updated, I think with ad_form you can only ensure this is true for one form.

The query-writer url api is designed to make it easy to handle insert/update/delete of any number of objects in one pass. I'm sure you can do the same with ad_form, but you don't need to write any tcl code with query-writer.

Query writer has flags for each attribute, arranged by group. By default there are Admin and 'registered user' groups. These flags could be used to build forms which only show the attributes (and even only allowed attribute values) on a per group basis. So one form could serve both admin and regular users.

There is nothing to prevent someone from extending query writer with a generic form. Given the object name, operation, and group, you can relatively easily do this. It might be nice if you could pass in a subset of possible fields you want displayed. Query writer already has help fields on a per attribute basis that could be included as well when errors occur.

It is just my opinion that a written down data model is the best documentation of what is possible with a package. Trying to discern/write the data model by reading/writing a bunch of tcl or pl code which creates a table attribute by attribute is impossible for my small brain. I've written and used more generic methods of representing a data model, but they quickly become useless when it is time to figure out how to use it. I also like being able to drop a package and recreate it to see if it installs after I have made changes. Probably you can do this with the cr too.

Most of the stuff I work on isn't what I would consider a content application. But if I needed revisions, and whatever else is on the laundry list of features provided by the cr, I would still look at using it. Query writer is not a content manager.

Hi Joel,

Great work!  I better update mine to use /lib instead of /resources.  Also make a /test :)

Great work!

I totally agree with Joel. I recently spent a whole day scratching my head over how ad_context_bar really worked, and why there seems to be almost half a dozen ways to do the context bar. In the end I did resort to the 'Wrong Way' :)

-Vamsee.

Somewhere in the aD literature (not that it means that its correct) is the argument that trying to make a piece of software serve every possible situation it's likely to encounter will exponentially increase the size and in the end will not 100% solve the problem.

How this applies to a toolkit of multiple intergrated modules, I don't know ...

actually I thought I submitted a context_bar wrapper for the family of context procs to take an option specifying the separator character which was then specified by a parameter.

I'll have to go dig that code up.