Forum OpenACS Development: XOTcl remoting services

There have recently been a couple of enticing announcements about
SOAP-based remoting solutions available to the OpenACS community,
namely TWiST by Tom Jackson and SOAP-DB by Rocael.

As many of you know from Stefan's presentation in Boston, we are doing
as well a substantial development at our institute in Vienna. We do
not want to fall into a "mine is better" discussion, but emphasize the
architectural approach we are following and sketch here the provided
services of mostly Stefan's SOAP and ORB implementation. Therefore, we
try to give a broader picture of the developed (and developing) XOTcl
based remoting services.

Motivation:

These XOTcl based remoting services are not meant to reproduce the
"classic OpenACS scenario" of proposing "just-another" package to the
community, as pointed out correctly by Don, but the intention is to
develop a more generic and framework-integrated solution (not
necessarily ours), capable to work with several
marshaling/demarshaling packages (not necessarily ours) and naming
services.

Towards a remoting architecture for OpenACS

Stefan is developing a generic remoting framework for OpenACS / .LRN
as part of his thesis for more than a year now. This project started
with a thorough review of the existing "half a dozen" OpenACS based
remoting packages (as Andrew Piskorski points out in
http://www.openacs.org/forums/message-view?message_id=316610), in
particular soap-gateway, the tsoap port and the XML-RPC package. With
that input and the observation, that there is an an obvious
correspondence between WSDL and Service Contracts, it became clear
that we have in OpenACS a good chance to come up with a highly
generic solution, which could make the framework attractive
for highly flexible middleware applications. As we are academics,
our profession is to investigate and develop new approaches with
the potential for publication.

Stefan's work currently contains two parts:

• XOTcl-request-broker: a CORBA-style request broker (aka xorb).
  the request broker as a plug-in infrastructure for various
  remoting protocols (SOAP, XML-RPC, ...)
• xotcl-soap: is the currently most dominantly used service

What is a request broker (xorb) all about?

Procedure call abstraction is a well-established concept in OpenACS
through service contracts. Request brokerage is available in many
different middleware flavors, e.g. CORBA, .NET Remoting, Web Services (in their
SOAP variant), DCOM, Java RMI etc. The developed request broker is
basically designed as a generic object-oriented broker for abstract
procedure calls, i.e. local and remote ones.

Request brokerage is a generic form of request redirection, which is
fairly common in OO frameworks. XOTcl itself provides several novel
means for providing chains of redirectors (e.g. through mixins). One
research goal is to rethink these redirection approaches in terms of
remoting and middleware. A part of this discussion came up in
(http://www.openacs.org/forums/message-view?message_id=588213).
Note that functions/methods can be in general called explicitly,
or via URLs (the request processor) or via remoting services.

Publishing services by annotations:

OpenACS supports declaring ad_procs as public or private. Similar to
this, XOTcl supports the same flags. The declarations introduced via
the remoting infrastructure through mixins allows to go one step
further and to publish methods for remoting purposes through
annotations. This allows great reuse of code and supports all OO
features such as sub-classing (specialization) etc., such one can
provide a customized remoting services, where everything is the same
as in a more generic one except a few differences.

Role of Service Contracts

Service contracts - as they are available in OpenACS today - are a
very powerful construct somewhat similar to service announcements in
middleware and software-bus architectures, but they are not developer
friendly (as they break locality) and they are not compatible with the
needs of todays remoting SOAs. The request broker addresses these
issues and provides an easier to use and more local interface to
service contracts based on XOTcl objects. These service definitions
can be transformed automatically into WSDL and can be more easily
published.

The overall idea we are trying to follow is to enrich existing
concepts rather than to replace them. Object orientation in terms of
XOTcl is not meant to provide a new language to replace existing TCL
code, but rather to deliver advantages of OO abstractions to the
OpenACS developers. It is fully compatible to existing TCL code. The
ideas of extending service contracts for remoting applications is
analogous. Essentially, one can replace occurrences of acs::sc::invoke
uses by its xorb counterparts.

The current features of Stefan's packages include:

• support for Tcl procs and XOTcl classes and objects
• server and client-side implementation of SOAP 1.1
• architectural support for maintaining state
  between invocations (one might call a specific
  instance of an object (e.g. a certain shopping basket)
  and modify it)
• publishing interface (including dynamic WSDL 1.1 generation,
  based on service contracts)
• SOAP 1.1 support (SOAP marshaller / demarshaller on top of tdom)
• Modular / extensible request handling: xosoap allows for registering
  interceptors / handlers that add functionality (logging,
  authentication/ authorization, caching, custom) for various scopes
  (per-service, explicit sequencing, ...)

Status:

The framework is successfully deployed in the context of European
Research Projects (www.icamp.eu, www.prolixproject.org) and is
currently evaluated for EU4ALL by Emma.

Although the mentioned features are fully functioning, changes in the
naming of the concepts, or in the composition of the components are
not unlikely. For example, in the icamp project, we are developing a
lors mediator interfacing with a SQI based repositories for federated
search of learning resources. Queries are distributed via SOAP to
different servers, the results are collected and transformed to RSS
and delivered to the asking client. In order to work in larger
quantities, we had to use asynchronous queries (asking simultanueos n
repositories, not blocking connection threads), which lead to strange
problems in aolserver with vwait. Now we have rewritten the basic
communication infrastructure, and things work smoothly. However, the
handling of asynchronous queries and the new basic infrastructure lead
to some interface changes as well. So, we want to gain more experience
in larger applications before we release the package to the general
audience.

What remains to be done

• more thorough test cases (currently just about 100 tests)
• Complete compliance tests with SOAPBuilder Interoperability Lab
  (20% done)
• as always after larger changes, code cleanup
• Enhance and test publishing mechanism
• Legacy test: replace ALL acs::sc::invoke uses by its xorb equivalent
• Polishing consumer-side stub creation
• Rewrite support for XSD complex data types based on XOTcl slots
  (50% done)
• Provide for WS-I compliance of WSDL generation
• Updating documentation and providing more examples

We should be able to provide a public prerelase withing the next weeks (early Mach)
and organize a tutorial at the Vienna OpenACS conference, if there is interest.

First, I want to make sure that there is no misunderstanding: We certainly do call for a more generic and viable solution in terms of call abstraction & remoting for OpenACS. However, this is not necessarily to be found in our work alone! All I want to stress at this point (before we can play our card = upcoming prerelease) is that any decision (either by the single developer or the OCT) should not be limited to a rough "SOAP-goes-Core".

ad-1 :: published API & code stability

As for the API, this is -- as prominently mentioned by Gustaf Neumann -- our top priority for the prerelease. As long as you refer by "code stability" to a stable interface, we certainly have to meet this fundamental requirement. However, the interface is already matured (though still not stable) through incorporating the experiences from the projects mentioned above and beyond. Moreover, the interface will certainly be familiar to many of you as we tried to stick with established notational forms, for instance, when declaring a contract the OOish interface resembles the good old arraylist.

ad-2 :: oracle support

I cannot comment on the fundamental issue here, nor the overall opinion in the community. Currently, the entire request broker operates on the established collection of db calls offered by acs-service-contract with only minor extensions. However, there is currently not a fundamental rewrite at this level (which might though be a good thing to do at some point) and therefore the Oracle issue should not be a barrier in this respect.

One intention of on our report above was to contribute to a more structured discussion on the state of (a)
abstract dispatch mechanisms in OpenACS and (b) of remoting support for OpenACS.

As a first step, we might come up with a state-of-the-art report (xowiki-based), requirements gathering based upon the results of the existing solutions, review of available code, ... we can then distil this into a consumable comparison scheme.

Again, this would also be an interesting task for a thematic workshop during scheduled Vienna meeting, such a comparison being the joint work artefact.