Hi All,

I just wanted to announce that we (]project-open[) are going to publish very soon a generic XML-RPC bridge for the database API. Please find the details below and let us know what you think. We've are still in an advanced planning phase, so there is still some room to influence our design. The package will be licensed under GPL V2.0.



SOA:

There is an incredible hype around this little acronyme ("Service Oriented Architecture") these days. Both SAP and Oracle are rewriting (or at least pretending to...) their applications to support this concept.

To me SOA means a kind of "component architecture" for web services (WS). You have "components" (WS) and you've got "glue code" (scripts, XML routers, Workflows, MS-BizTalk, ...) that connects the different WSs amongst each other and impose a certain business logic and some sequencing.

So given this definition (please let me know if you think I got it wrong), we'd propose nothing less then a complete "SOAization" of OpenACS / ]project-open[. Using this interface, any external application with the right credentials would be able to integrate with and "remote-control" an OpenACS instance.

SOA might imply SOAP as a communication protocol (Does it? It's become just such a fuzzy term), so somebody might argue that our XML-RPC-based approach isn't really SOA compatible. However, we could easily create a SOAP-version of our generic interface in the future. Any opinions?



Interface Requirements:

During some recent customer surveys it turned out that "integratability" was #1 inhibitor for our potential ]project-open[ customers. Our customers frequently have legacy systems written in MS-Access etc. that need to read from and write to ]po[. So they are preoccupied by the integrated approach of ]po[ and need to be reassured that there applications will continue to be supported.

On the other hand there is usually very little budget available while most of our customers are not very "IT-sophisticated". So the integration needs to be easy and transparent (easy to debug).



Interface Generic Architecture:

Normally we'd need hunderts of different XML-RPC calls in order to access and modify the hundereds of object types in OpenACS / ]project-open[. However, after some discussions we've come up with the idea that OpenACS is really just a database (objects "live" in the DB, instead of "living" in the memory). So we'd only need four different API calls to access any type of object:

- "LOGIN": Authenticate and generate a session key

- "SELECT": Retreive the fields of an object from the database

- "UPDATE": Modify the fields of an object

- "PLPGSQL": Execute a object__new() or object__delete() procedure to create and delete objects

The integration with our "DynFields" (formerly known as "FlexBase") metadata model allows us additionally to access fields in "extension tables". For example we've got a table "im_employees" in ]project-open[ that contains payroll information for users of type "employee". Fields in this table are automatically selected and updated with the XML-RPC calls.



Interface Communication Protocol:

We've been looking at all three options for the interface. We have finally decided to go with XML-RPC, because it is the _easiest_. Here are the details:

- XML-RPC is the oldest XML-based interface standard. It's been kind of deprecated with respect to SOAP in the last years. However, it's much easier to use then full-blown SOAP.

- SOAP is the standard today for XML-based communication. In particular there is the cool WSDL feature that allows yout to meta-describe your communication protocol in XML, so that the usual tools (MS, IBM, ...) can automatically create proxies/interface objects for the SOAP messages.

- REST is a cool newcomer these days. The main difference to SOAP is that related objects are _reference_ (included as URIs in the XML text) instead of _embedded_ (included inside the XML document as subobjects). In REST every object (user, forum item, ...) has its on URI (a slightly generalized type of URL).


Interface Security:

We're sticking with HTTP over SSL ("HTTPS") for security. For authentication we're probably going to use a session key that is generated using the "LOGIN" message.



Dependencies with ]project-open[:

We'll try to keep the XML-RCP-Bridge free from dependencies to ]po[.
The main issues is the "DynFields" SQL Metadata system which hasn't been included in OpenACS yet. However, we've mainly going to use the table "acs_attributes" to extract information about "extension tables" and dynamic attributes, so you could edit the contents of this table manually without DynFields.

As said in the into, we've already taken some decisions (XML-RPC and the generic interface idea), but everything else is still flexible.


Cheers,
Frank

frank_dot_bergmann@project_dash_open.com
http://www.project-open.com/

Hi Frank,

I've been considering a general wsrp service also.

As I see it,

1. the service should use openacs authenticate and permissions (via http/https) before executing a call and/or screen based on ip and a hash, etc etc.

2. the call should be to a tcl API instead of db_* so that values can be easily customized to meet requirements that would complicate db_*, for example to allow general audit features to work (since ecommerce packages use them and this is wsrp).

3. there would be a table mapping "allowed" calls to specific protocol access and openacs tcl api (all other requests are refused and optionally logged).

4. there would be different urls within the package for different protocols, for example: soap, xml, cgi, and other third party standards (such as legacy ones). I think this approach would largely prevent the splintering of services into innumerable packages, as is possible with OpenACS on the client side, at least when using payment-gateway which requires another package for handling each protocol (currently either ezic-gateway or authorize-gateway only).

5. the package would be.. package aware so it could be mounted any number of times in any number of places and provide different services to each.

I'm also open to suggestions..

Hi,

Just a few answers/comments:

> Install the PostgreSQL ODBC driver

This is basicly the idea of the interface: Document the PostgreSQL API once and use the documentation for both, direct ODBC (LAN) and WS (Internet) access. This is why I've called the package a "bridge" (-> OO bridge pattern).


> run any query through SOAP

Hi Roc, that sounds very interesting. Could you provide us with the source code? It would be nice if we could offer both SOAP and XML-RPC. I'm not sure whether we'd integrate it "Phase 1", but we'd definitely adapt the design of the package in order to be extensible.


> 1. the service should use openacs authenticate and
> permissions (via http/https) before executing a
> call and/or screen based on ip and a hash, etc etc.

Hi Torben, that was our idea as well. There is a security token scheme from some bulk mail package from Malte Sussdorff that we'd use. So the user would authenticate once against OpenACS auth, obtain a hashed auth token and execute multiple calls with this token. The only issue with this is that the token has a unlimited livetime today. We might have to change that...


> 2. the call should be to a tcl API instead of db_* so
> that values can be easily customized to meet
> requirements that would complicate db_*, for
> example to allow general audit features to work
> (since ecommerce packages use them and this is wsrp).

Right, we're using the TCL API already for our prototype. Also, we need to extract fields from "extension tables" as defined in acs_attributes, so we had to discard early the use of the db_* interface.

But I don't understand the "audit" thing and how that relates to ecommerce. Could you elaborate, please?

> 3. there would be a table mapping "allowed" calls
> to specific protocol access and openacs tcl api
> (all other requests are refused and optionally logged).

Do you refer to introducing permissions, similar to the ones of the PostgreSQL database user? Maybe read and write permissions per object? That's an interesting idea. We've only been considering a all-or-nothing scheme until now.


> 4. there would be different urls within the package
> for different protocols, for example: soap, xml, cgi,
> and other third party standards (such as legacy ones).

Right, that was our idea.

> prevent the splintering of services into innumerable
> packages

I think I understand. So the extensibility towards SOAP would probably be the first step towards such a "unified" architecture, so we're back with Rocael and his SOAP interface.

Thanks a lot for the feedback so far, I really appreciate it.
Frank

frank_dot_bergmann@project_dash_open.com
http://www.project-open.com/

Frank, you write:

"..The only issue with this [item 1, authentication] is that the token has a unlimited livetime today. We might have to change that..."

I imagine implementing a hash that works similar to how the session cookies work, where a component of the hash requires re-authenticating after a certain period and it ties in to existing session procs.

Since this is between machines, maybe it could use the shortest of the session intervals, SessionRenew (or a combination of SessionRenew and SessionTImeout ) in the kernel parameters etc.

Anyway, I'm not certain of the details beyond this.

Our forthcoming SOAP / WSDL package will expose arbitrary Tcl procs to external SOAP invocation.

Will be released some time soon..

Frank, your arguments and plans sound basically reasonable to me. Beyond that, I haven't studied these issues enough to have any more ideas. Since obviously you have, I'll just assume you're on the right track. ;)

we'll commit our soap DB stuff this week during the bug-stomp!

We have our stuff working for months at our production site, will be good if other take a look a see if they wanna use it. We'll happy to support and use an official tool.
Anyway, ours is relatively simple and focused on just passing the queries to the other side and get results back!!!

We wouldn't even have started with our own stuff here if there would have been some previous work. So yes, that's a pitty. But maybe "competition" is rising the spirits (and code quality) on all ends?

However, SOAP and XML-RPC are very different protocols and known web services such as the one from Google support both protocols, and even REST as a third one. Also, there may be differences in the priorities and scope. For example, we had to "reinvent" full-text search and filestorage for ]project-open[, because the existing stuff was completely unsuitable for us.

Another option is to keep our stuff private and sell it as a proprietary package. I've got no problem with that either... :--)

So Roc, I'm really curious to see your code. Could you please send me the current version as soon as possible? Maybe we can still stop the development here and adapt your stuff.

Bests,
Frank

frank_dot_bergmann@project_dash_open.com
http://www.project-open.com/

May I wish that the OCT takes a look at the *features* and says which it likes best and then hopefully between the three of you, you could mould it into one nice and neat package, that suits all your needs?

Ok, the package is called:
soap-db, is on cvs now.
Basically it allows you to perform queries in a db_ style, send it to a soap server, and operate in the callers environment the results in the same way you do with the normal db_ API.

soap-db is the client API that mimics the db_ API.
This allow you to connect to any database without the need of having specific drivers for aolserver.

Also, there is a server API implementation for openacs as well, the working code we use is here:
http://cvs.openacs.org/cvs/openacs-4/packages/soap-db/tcl/example-soap-procs.txt?rev=1.1.1.1&view=auto

You need to have installed tsoap package, plus for the server side you'll need the soap-gateway (which allows you to expose oacs procs to soap).

The original design was mine, the development was done by derick at galileo.

We have been using it in production for quite a while without troubles.
The only thing that we cannot handle yet is transactions (like db_transaction), the best approach we use is to substitute that to a complete SOAP proc call that in the server side the proc should handle itself the transaction.

The package is oacs centric in the sense that it runs queries through soap and expects a return in the same form that the db_ API will return it internally, therefore you can use stuff like <multiple>> and list-builder in your site even running queries through soap.

Although we don't have non-oacs examples for the server API (the one that will be running the queries, and returning them in a db_ way), I'm pretty sure you can build it any soap server-language environment.

Fine-grain security policies was not the goal of this package, first due our project goals, that was to run queries in this way, within an internal-secure-restricted network. Since this package does what it is meant to do, you might be able to build a top of it the security policies, in the same way that db_ API does not inherently check them but other procs does.

Hi Jade!

Is there already a test version of your SOAP/WSDL Client-Server? I need a decent SOAP implementation in the next weeks and my options so far are TclSOAP or, preferably tsoap or java. But I'd like to see your package as well.

Wolfgang,

Jade asked me to report the status of our project, tentatively called tWSDL.

The first thing to keep in mind is that this project is completely independent of OACS, maybe more important: there are no additional dependencies beyond using AOLserver and tDOM. Parts of it run with tclsh, most runs with libnsd loaded, the client side will likely be completely independent of AOLserver (maybe not independent of libnsd).

WSDL is a step beyond XML-RPC and SOAP. WSDL is more like a data model for a service. An important new feature is that WSDL defines message types. By defining types, WSDL enables type checking prior to invoking a procedure. The same information can be used to figure out how to construct messages, where to send them, and how to validate the result.

Unfortunately Tcl doesn't specify types, so there is a little more work required if you want very good type checking. On the other hand, Tcl's very simple (and few) data structures makes it easy to model a procedure call and result in XML Schema.

Speaking in OACS language: WSDL is like a page contract, where you can equate a page to an WSDL operation.

WSDL could potentially cover a huge number of implimentations. However, tWSDL is first restricted by the WS-I Basic Profile. This means that the basic protocol is SOAP/HTTP using document/literal, plus a few hundred other restrictions. (Literal is required, tWSDL may end up supporting rpc/literal). These choices greatly reduce the complexity of XML processing. Third parties can add their own protocols, by adding a routine which processes a small slice of the request. Note that collections of operations, called 'portTypes', are completely independent of the protocol, so everything below portTypes: operations, messages and types never change if you add a protocol.

So what do we want to be able to do with tWSDL?

Server side:

- It should be possible for the developer who knows zero about WSDL/XML/SOAP to easily create a service based upon a Tcl procedure.
- It should also be relatively easy to create a service which enforces type checking on the input and output documents.
- More advanced developers should be able to construct services which can process and produce XML documents at the complexity of, for instance, an invoice or sales transaction.

Client side:

- A tWSDL client should be able to read a WSDL file and create a client Tcl API.
- Client code can then use the new client API to invoke the service.
- In the simple case, a single call to a client API will return a Tcl result.
- In more complex cases, client API's will construct an XML document, then invoke.
- Client will have full ability to validate outgoing and incoming documents and report faults.

So where is tWSDL right now?

The server side is progressing very well. We can create a service based upon any Tcl procedure which produces a simple result (list).

Extending this to more complex results will be relatively easy. Although there are automatic procedures for handling the usual case, these can be replaced with custom code. Eventually there will be helper procedures to make it easier to write custom code.

It is important to point out that custom code doesn't need to deal with any of the other parts of tWSDL. Essentially the custom code which invokes the procedure is handed a Tcl representation of the XML document (by reference). This document could be of any complexity. The custom code completely handles this processing until it returns a reference to the result XML. In the current API, this 'custom' code is auto-generated.

The separation of the invoking code from the service ensures that the developer is not limited by 'my' preconceived ideas: any XML document can be processed and/or returned.

Although simple things are simple, the simplist thing: no brainer WSDL, is still a few weeks away. This is nothing more than a wrapper with a few conventions on how things should be done. At the moment we have two services, one imaginary (no real backend), one real. These two services offer enough diversity to build the no brainer service.

Since tWSDL will be used with ACS code, I will develop a switch for ad_proc, which Jade mentioned above. The switch will employ the conventions (hmmm, maybe we can configure those), to build a service at runtime. In any case the switch will do nothing more than cause a single proc to run which will use introspection to create a no brainer service. It should be just as easy to call the proc outside of ad_proc (at the correct time).

Another important part of tWSDL is fault/error handling. Currently client errors are handled: incorrect target namespace and invalid document (type errors). Server errors are not handled yet, but will most likely return a stacktrace. You still get a 500 response, which is all most clients need to know that it is a server error. Error checking/handling is done according to the WS-I Basic Profile.

Code Release:

I'm open to suggestions here. I don't mind releasing this package now, but anyone using this needs to remember that major changes could result. However, I have already done a handfull of major changes and the code changes to the application code have been easy and fast to apply. Application code relies mostly on metadata, so there is very little risk that a big change in implimentation would change the amount and type of metadata required. Most changes so far have been to the form in which the metadata is presented to the API.

OACS has a unique and valuable API, there is also a huge investment in page flows which cannot be replicated by a web service. A web service can potentially be useful in OACS for existing and potential packages, but not every package. The only thing that is really universal across all OACS installations is the Tcl API. tWSDL offers the potential to expose any Tcl API as a web service. In addition, it should be noted that OACS also uses conventions related to types and data structures. It would not be difficult to exploit this metadata to build high quality web services with very little additional effort by the developer.

There are still many questions to be resolved for any web service package including security and how a web service should fit into the request processor. The fact that web services use a different processing model than a pure HTTP service makes this a difficult question.

It is more likely to work well with services defined by .NET, as (unbelievably) MS generates very high quality WSDL files. tWSDL will generate a WSDL file to match .NET, but this work is not complete yet. I still need to add the 'types' element. The client will read the WSDL file to generate the API, but it needs to be compliant with what the tWSDL server would generate. At the moment, I'm using an XSL transform on the WSDL file. Someone could also create an XSLT to generate the server side application in Tcl, all that would be left is to write the Tcl procedure, and maybe any special type validators.

I'll try to find a place to put the current documentation on line, and figure out how to get a current snapshot of the code. We are using svn instead of cvs.

Hi Wolfgang:

The server portion is in a working state, and we're doing work to integrate it into our own implementation of ad_proc (mostly done).

We'd like to do a similar example integration with OpenACS when we release it, so that a lot of the complexity is hidden from the developer. We'd prefer that when people do the first look at it, their experience is "wow, that's easy" rather than seeing too much complexity (which is inherit in WSDL).

But if you're willing to deal with the underlying API, we can release an early, pre-release version.

Otherwise, I expect we'll have something ready within the next few weeks.

Here it is:

http://code.google.com/p/twsdl/

Wolfgang,

You can now browse the current code at: http://twsdl.googlecode.com/svn/trunk/.

Or do a svn checkout:

svn checkout http://twsdl.googlecode.com/svn/trunk/ twsdl