Forum OpenACS Development: Re: The future of OpenACS

Rewriting in XoTCL does not make any sense to me either. Why change a running system to make it run slightly better yet radically different with even less documentation for the developer to go around than there is for OpenACS per se.

This said, I love XoTCL and what it did for my application development. And more of it will popup in OpenACS CVS in the medium time for new functions. But I would not go and rewrite e.g. Forums in XoTCL unless there is a compelling reason to do so (read: customer requirement).

What would in my opinion be killer apps for OpenACS are things which haven't been discussed here:

WEBSERVICES

TWiST works great and I am sure Stefans code works beautifully as well. Yet we need to bring them into OpenACS properly and devise methods where an administrator can quickly say: "I would like this information available as a webservice". Ideas on that:

-- Make a form a webservice. If we use ad_form, provide a switch that, if querying a URL with a certain parameter gives you a webservice for the form.
-- Enhance listbuilder to return WSDL formatted XML in addition to CSV

I am aware that I will get lot's of bashing about this for not understanding the concepts of Webservices, but what it boils down to is:

"I need the information on this table available in my financial data system"

"We need to interface our user creation with yours to keep it in sync"

"We have a Yahoo Widget which wants to add a new task"

If OpenACS as a toolkit provides interfaces OUT OF THE BOX, without the developer having to do anything about it, then we talk killer app. Oh, and yes, we would need to have security on the webservices and lots and lots of other things to take into account, above is just a suggestion.

What can be done? Have the most experienced people with regards to webservices (Stefan and Tom) along with interested OCT members come up with a generic strategy how to support webservices in OpenACS. Get "customers" like Quest and ]po[ and E4 involved to state what they need and see if the model suits them. Then collaborate together to come up with the funding for a generic solution to be put into OpenACS. But as there a customers with needs I have the slight suspicion there will be money as well (or at least development help).

EXTEND FIELDS

I think the reason that we have 5 (or more) ways of dealing with dynamic extension of data fields shows me that there is huge demand, great intellect and capacity for building it and ZERO coordination to make this a killer feature for OpenACS. But this is what customers want.

They want to be able in their application to extend the fields a user has to fill out when signing up, WITHOUT hiring a developer to make that happen.

They want to store their custom datafields for projects and show them in a project overview.

Why is it at the moment impossible to do this with OpenACS?

If the most common needs for a user of OpenACS can only be achieved by hiring a developer who is knowledgeable in OpenACS, chances for him using the toolkit are slim, because he can use a LAMP application and hire a smart PHP kid from Ukraine for $20 per hour to make the changes for him.

What can be done about it? Collaborate on https://openacs.org/xowiki/dyntypefields by writing up what we want to have and pointing to bits and pieces of code that already provides this. Then ask who can work on achieving this and then have those who can work on it vote on which existing solution to base the new one (or to write a different one from scratch).

DOCUMENTATION

Our documentation sucks and the one for the packages is even worse (XoWIKI becoming an exception to the norm). But without documentation on how to achieve certain things in OpenACS we will not be able to attract new users. And if our goal "only" is to aquire new developers who love the toolkit, than at least the developer documentation needs to be up to date. We (Anett and I) have worked over the last months to clean up https://openacs.org/test-doc and enhance it bit by bit with new information (mainly get an update for TCL for Dummies there), yet this showed us how MUCH we are out of date and how difficult this job to update the information is going to be. And to make matters worse, it probably needs the best people we have (in terms of development experience) to write this documentation to attract new developers, which is a catch 22 as they are usually in high demand (due to the fact that the documentation sucks so badly that noone else can quickly learn this).

What can be done about it? Among the companies in need for new developers come up with a "fund" for the outcome to have a developer tutorial along with updated Problem Sets and training / teaching instructions. They need this anyway if they want to grow, so they might just as well shoulder the burden and not do the same effort each.

Then try to get some EU or other funding to partly cover the writing of documentation for packages used in the funded project and have them written. There are excellent writers for documentation out there and I am fairly sure you can point them to our application and as long as you are open for questions, they can write up (and screencast) what they have been doing on the application, without having used it for a couple of years. And yes, this obviously depends on the level of complexity of the application.

ATTRACT NEW DEVELOPERS

We have a lot of universities using .LRN. They have in house staff that are knowledgeable with the solution. They have students who might need to learn about developing these kind of solutions, so why not teach them? As Gustaf mentioned, a lot of volunteer effort's benefits come from the acknowledgment of your work. So why not invite OCT members to be guest lecturing a course at your institution? This would already get into the direction to providing the tutorial as discussed above and if the lecture is taped and edited it can then be used as a Web based Training. I am not an expert in this, but with all the effort going into LORSM and IMS LD, I assume we have enough knowledge in the community to achieve this.

Why not providing/ exposing functionality and data over different protocol channels (SOAP and XML-RPC), without the need of multiplying the coding effort (e.g.,rewriting the serving bits of code, interface descriptions)? In fact, multi-protocol support is something rather new in framework design, most established frameworks are rather short-handed in this respect. Note, by multi-protocol i do not simply mean providing an xml-rpc and soap package and having the individual developer deal with exposing their code to them. There should be a certain protocol transparency from the developer's perspective. I don't want to give examples from literature, but just a hands-on example: Ruby on Rails comes with a somewhat multi-protocol support (at a rather low level, for my taste) in its Action Web Service module. By switching around some parameters, you either go for an XML-RPC or SOAP call/provision, through a common interface.

Everything should be automatic! This is like ASP.NET, just add a few decorations to you code and it becomes a web service...or not. Actually if you want to have complex types or validate the input you are totally screwed, now you need to write your own code for doing this. Apache AXIS is worse. Although it _is_ true that once the bits get to the protocol level the developer doesn't need to know anything, but by that time who cares? There are no switches to throw anywhere in any system, and you still lack the most important ingredient: a published description of the service and how to access it. Maybe somebody will recognize the importance of following a standard so that external clients can actually use your interface.

Now, what are the issues with OpenACS? The main issue is that most OpenACS applications depend upon page views. By that I mean that pages serve as a procedure. This makes it nearly impossible to reuse this code by a different protocol. For instance, my query-writer package requires all form submissions to go through a special page, very similar to rpc, minus the xml. Until I write a procedure, or set of procedures which covers this functionality, no amount of developer support will help me.

Another issue specific to Tcl is important to recognize: Tcl does not maintain type information, ad_proc actually complicates the situation, and we know nothing about certain things like return types, use of uplevel/upvar, etc. Many programming languages know all this information and so it is 'easy' to serialize a data structure to XML. This is why there are so many rpc packages in every language. But rpc has many problems which were never solved because they are much harder. First problem is that server and client really need to use the same language for full interoperability. Otherwise, either one or the other needs to learn to handle conversions. Second is that it is unlikely that anyone anymore really wants to expose an internal procedure to the network. The best method for handling this is to write a wrapper procedure to restrict use or perform any additional checks that usually take place prior to execution. One of these 'checks' is validation. This is where the wheels fall off of rpc and it is why nobody, not .NET or AXIS performs validation out of the box! And by validation, in this situation I only mean simple validation like is 4 an int? Derived types and complex type validation...this isn't even supported if you wished to do it. Just make up you own method of figuring out if the input is valid, test, debug and hope.

So if the big boys are not offering this as a feature of their products, with billions spent on development, maybe it isn't so easy. Configuring a procedure as a remote application, or whatever it is called, will always require a certain amount of information. How to collect, store and expose this information so it can be reused by new protocols, maybe at the same time...that would be helping the developer.

Hmmm maybe look at tWSDL. tWSDL impliments a single protocol: SOAP with document/literal. What is probably not appreciated is that the protocol is just a few pages of code, and is chosen based upon the client request. If someone were to write their own 'binding' and protocol handler they could reuse the entire tWSDL code base. The service would be fully described by the WSDL, input messages would be validated prior to use. If XML was not used as the representation, then the protocol handler would use some other method besides tDOM to deserialize and some other method to serialize the result. Writing the code is the least time consuming part of this. Deciding what your protocol is and how it works will take the most time.

One thing has become clear over the last few years: developers are starting to recognize that interoperability is the key. Interoperability is achieved by restricting the number of options and requiring public descriptions of services which don't require out of channel communications.

A note about code generation from WSDL. TWiST does this, not just stubs creation like AJAX. But it is only for the server. I have begun work on a TWiST Client. The client can run along with the server, but it will very likely not require AOLserver, that is it will be pure Tcl. But I'm developing it against Google's AdSense service. Everything is https, so I'm using ns_https to pull in the WSDL to read. The service developed on AXIS and uses a feature not in tWSDL/TWiST: SOAP headers. Anyway, the goal is a client which is created from reading the WSDL with API creation for each operation. Creating a document and sending it should work just like in the TWiST server. The return document will be available, but there will likely be a series of generic procedures for turning the document (as a Tcl representation) into Tcl structures, lists, arrays, etc. Of course there will be optional or automatic validation on the return document.

Working with a procedure defined via ad_proc actually presents very similar issues as an application page. ad_proc serves many purposes, but the way it works makes it harder to expose as a web service. This is why TWiST works the way it does: configuration is done via <ws>proc, which provides a wrapper for the internal API. Inside this wrapper, you can specify any amount of code. Maybe you could source a tcl page and do something with the resultant state. Whatever goes on inside the wrapper, you just need to return a list which represents the output message.

One point about typing with TWiST: more important is the structural typing, such as the names and order of the complexType elements, how many of each of the elements, can it be missing, is there a default, if missing, does this indicate it is 'nil', etc. Otherwise, your return document will just be a smashed together mess, but internally TWiST does not care about types, just structure. Essentially the type information is a description of the template: what is the document going to look like. Nobody would ever suggest we just run a tcl script and return a list of all the variables, the results need to be formatted, and this is what the type specification does: defines the non-data parts of the document. The wrapper code doesn't need to know anything about XML or any of this configuration, it just needs to return a list which will be formatted into a document.

When the TWiST client is done it will only handle document/literal, whatever is available now is in the wsclient package of tWSDL. I think everything is in a web page, and it requires the ssl module to be installed, at least to use ns_https.