Forum OpenACS Q&A: The whole java thing, Tclblend and Webmail

I've been investigating porting some of the java stuff coming out of aD, and as a test case I decided to port webmail which relies heavily on oracles sqlj java implementation. I'm to the point where I can now able to send and receive emails without attachments using the webmail interface. Attachments dont' work yet, since I'm only able to pull lob data out of the database, but with a little more work, I should be able to get it so I can put lob data back into the database through the java JDBC interface to postgres.

The webmail module consists of two main parts that execute java code inside of oracle. One is a message parser that reads in messages from qmail performs mime decoding, and the other is a message composer that performs mime encoding.

I was able to reuse most of the java code without modification. The only exception being that sql statements that are declared using a c-preprocessor looking syntax that starts each sql statement with #sql. Why oracle chose to do it that way is beyond me. Why not for instance, just extend the java sql classes that are already specified in the product api? The postgres JDBC interface is handled this way, and the result is much cleaner.

The acs implementation of message composition went as follows:

1. Stuff message headers and body into the database.
2. Call java function to perform mime encoding.
3. Pull mime encoded message out of the database and send it using qmail-inject.

The ported implementation works as follows:

1. Stuff message headers and body into the database.
2. Call java function to perform mime encoding via exec function in tcl. java talks to postgres using JDBC interface.
3. Pull mime encoded message out of the database and send using qmail-inject.

The message parsing transformation is similar. The oracle/acs implementation is as follows:

1. Oracle schedules java process to check the qmail rx queue once a minute.
2. New messages are mime decoded and stuffed into the database when scheduled process runs.
3. User sees new messages when /webmail url is reloaded.

The ported version works as follows:

1. Aolserver schedules process to check qmail queue once every two minutes.
2. New messages are mime decoded and stuffed into the database using JDBC interface when scheduled process is run.
3. User sees new messages when /webmail url is reloaded.

Java is definitely not suited for web development. I've never been a big fan of TCL, but I now find it to be a dream compared to doing this stuff in java. I found that when writing java code, I spent half my time scanning the api library docs trying to figure out to do something that I would have easily done in TCL.

Tclblend seems to be a non-starter as it requires the tcl library to be compiled as a shared library, and aolserver compiles tcl as a static library. I'm guessing that aolserver uses static libraries for reasons of thread safety and speed, and I wouldn't want to start messing with custom compilations of either aolserver or tclblend.

Given that aolserver and java both are multi-threaded and have good support for sockets, a better approach might be to start up a standalone java application that talks to postgres through JDBC, and is controlled by aolserver via a socket connection. Doing it this way would be more portable and easier for users to install. The only extra requirement being that the user install java and compile the JDBC interface.

Of course, if aD gets too carried away with this java inside of oracle thing we'll just have to fork from acs, and provide similar functionality using tcl/postgres.

Dan, this is a *great* exploration of our possibilities. I've done the Java web development thing, and I agree with you that it is absolutely horrendous. However, if aD writes the Java, and we just have to change the SQL, it's not that big a deal.

What concerns me deeply, though, is the fact the interface is through process execution. This is a very bad solution from a scalability standpoint, because each call spawns a process which spawns a Java VM. That means that if there are 10 active connections, you have 10 parallel Java VMs that get created and killed, and that's pretty bad. That's as bad as CGI scripts each launching a Java VM (eek!).

I'm not downplaying your work, what you've done is 90% of what we need, and I'm psyched! The difference is that we need to have, as you mention, a standalone Java VM constantly running, accepting some kind of communication from AOLserver in a threaded manner.

Since Tcl Blend doesn't do it, we should go ahead and look into JNI and the C interface that allows us to launch a JVM and maybe interface that from an AOLserver module. I see no reason why this wouldn't work, but it should be investigated more.

This is great progress.

I would like to emphasize that this is just a proof of concept for using java in openACS devlopment and not a final solution for porting webmail. Most of the work on this project has involved adding java classes to replace the functionality lost by converting the sqlj statements. Yes it's true that mime coding/decoding would probably be better handled in TCL, but given the small number of people doing development/porting on openACS, it's easier for us to find ways to port the stuff from aD than to reinvent it.

I agree with Ben's comment about scalability, but for this phase of the development, I've been mainly concerned with the JDBC end of things and making the interface to the database functional in a generic way, so that future porting of java code will mainly involve rewriting the sql statements. In the short-term it would be easy to convert the message parsing side of webmail run as a standalone application that would run in a loop where it would sleep for some fixed amount of time, wake itself up and then process the mail queue, putting itself back to sleep at the end. The process could put logging information in the database each time it parsed the mail queue, and a watchdog process could be scheduled in aolserver to run periodically and check if the queue processing were working correctly, and if not, it could kill the current message parsing procees and restart it.

The message encoding is the main bottleneck in the current scheme, and it will require some investigation to work out a method for controlling a standalone java application in a robust and portable manner. Once I finish up the blob stuff and verify that I can send and recieve attachments, I will look into some of the possible alternatives.

The two main alternatives as I see it are to use JNI to embed a jvm in a loadable aolserver c-module, or to control a standalone jvm through a socket connection.

A JNI interface would probably provide a seamless interface to the java libraries, but the main problem with this approach would be that we would need to develop a loadable jvm c-module for aolserver. It seems like this would be doable, but I'm sure it's not a trivial exercise. Maybe somebody has already done this? I looked over at the aolserver site, and the only mention of java was the tomcat servlet engine, which from the sound of it, is not what we want.

The other alternative for interfacing to java would be to use a socket interface. Both aolserver and java have strong support for socket progamming, and I think it would be possible to provide a generic api in aolserver for issuing commands to a standalone jvm.

Anyway, I'm going to do some investigation on the two methods mentioned here, and if anybody out there has some other ideas, I would be willing to look into those also.

First - this is a tremendous coup, Dan.  I'm extremely impressed.

The fact that you got JDBC working with Postgres is a minor coup in itself, as the PG hacker's group gets a steady trickle of pleas for help from folks trying to do this.

JNI-in-module seems like the best way to go, by far.  It would be a valuable contribution to AOLserver, too, I'm sure other folks would want it.

A question about BLOBs.  Are you doing an emulation hack like I did or  using built-in LOs?  The built-ins aren't a viable solution as they can't be dumped, and secondarily because they make two files for each LO in your database directory.  Moving the hack I did for the driver into Java would be easy, and of course the SQL's already defined, you'd just use the same table the driver uses.

It looks like real large objects are going to be in PG 7.1 for sure.  Jan Wieck (who also did PL/pgSQL and referential integrity) has already committed his first cut to the PG development CVS tree.  He's doing a CLOB type at the moment in order to test the SQL and internal routines (it actually works a bit like my driver hack), so it can be tested without folks having to wait for him to hook it up to built-in types.

If he or someone else doesn't write character-encoded I/O for large byte arrays, I'll add it in, to make this built-in type pg_dumpable.

At that point we can drop my driver kludge and anything you come up with for Java.

The point is simply that you can view any solution you come up with as  being strictly a short-term thing, just like my driver hack.

Table lookup would be fine, but I wouldn't expect the standard uuencode stuff to run terribly slowly anyway, even in Java (I lifted it from uuencode myself).  I'm surprised someone would want to charge  money for such a trivial piece of code!

You might just write it up in Java and see just how slow it is.  Since  we only need it for a few months, we might not care.

Brent,

I tried to check out PyWX using anonymous cvs access, but I didnt' have any luck.  cvs said that it didn't recognize PyWX as a module name.  I downloaded the tar file but it seems that you've added quite a few files since the last release.

Okay -- it should now be possible to check it out of CVS
by following the directions posted on the sourceforge site.  It
is now in the "PyWX" directory, as it should be.

Have fun!

Dan,

We do wrap many pieces of the Tcl API, but that's mainly as a means to get things working quickly.  Furthermore, we are seeking a high level of Tcl integration, so some of what you are seeing are Tcl callbacks to Python, Python callbacks to Tcl, etc.

We plan on exporting all AOLserver API functions through Python, using the ns_python and pywx modules that are already present (in an early form) in PyWX.

The important thing to note is how the python interpreter is created and cached by AOLserver.  You will also need to devise a way of connection the Java I/O streams to AOLservers input and output buffers.  That's what the AOLbuffer C++ class does (in a messy way).

It might interest you to know that I'm getting started embedding a Scheme interpreter (Guile) in AOLserver.  It's not because I'm a
language-crazy fiend (I prefer to remember as few as possible), but
it dovetails with another effort I've got going on.  At any rate,
we might want to discuss embedding issues together so we can have a somewhat unified system of handling these projects.

I believe that once your Java interpreter is in-process with AOLserver, it's standard error will automatically be written to the logfile.  Then you need to decide if you want to redirect all I/O through AOLservers HTTP ports, or if you want to implement specific commands (like ns_write) that do this for you and leave standard I/O alone.

Anyway, let me know if I can answer any questions.

Dan,

Check out "Kawa" for a Scheme implementation on top of Java.  :-)

The Amaya web development kit embeds the Kaffe version of Java.  So we could probably look there for ideas as to how to go about doing so.

-Brent

I don't know which ACS functionalities this would break (if any) but you can disable abstract-urls entirely at your parameters/servicename.tcl:

ns_section ns/server/${server}/acs/abstract-url

        # enable abstract url handling?
        ns_param EnableAbstractURLsP 1

Just change "1" to "0".

Hi, anyone got any cool ideas about nsjava? Mainly, what would you use it for? I haven't touched it yet. But I understand that threaded Java chat servers are common, so my question is how could Tcl and Tcl access to the RDBMS benefit from having a Java VM in process near-by? Any neat ideas?