Forum OpenACS Development: API for parsing html-forms

Posted by Nima Mazloumi on 11/15/03 06:42 PM

Hi everybody,

is there a more convenient way to parse a html-page for it's form part? I used ns_http_read to get its content. I want to figure out the form action, the method (post,get) and the input fields it uses.

I tried to use regexp but it's not elegant and I have problems to extract multiple occurences of input fields :(

I really appreciate your help.

Best wishes,
Nima

2: Re: API for parsing html-forms (response to 1)

Posted by Tilmann Singer on 11/15/03 06:56 PM

A collection of non-elegant regexps for doing exactly this can be found in http://tclwebtest.sf.net .

When you are using aolserver 4 it is fairly easy to use tclwebtest from within with 'package require'. Otherwise it becomes tricky - one way I made it work is by calling tcl's 'source' before calling tclwebtest commands, but this is propably not very efficient.

With aolserver 4 you might want to use http://tcllib.sourceforge.net/doc/htmlparse.html instead.

Or use tdom - it has a html parsing mode, although it is not very forgiving on broken html, e.g. it crashes on things like non-closed tables etc. And I'm not sure if the html mode is available in the aolserver module.

3: Re: API for parsing html-forms (response to 1)

Posted by Nima Mazloumi on 11/15/03 07:09 PM

Thank you very much. Unfortunately I have AOLserver 3.3oacs1 installed - in accordance to the documentation. Why isn't the newest AOLServer 4.0 used anyway? Has this something to do with the modules needed for OpenACS? From your reply I understand that you use v4.0. Would you suggest to upgrade the server to 4.0? And where could I find more information on how I can do thatß

Thank you very much,
Nima

4: Re: API for parsing html-forms (response to 1)

Posted by Nima Mazloumi on 11/15/03 07:21 PM

For the Documentation it says that the following patches are needed. Did you patch v4.0?

AOLserver is available at aolserver.com
ArsDigita's AOLserver distribution (including internationalization patches, nscache, nsrewrite, nssha1 and the oracle driver) is available at arsdigita.com
The OpenACS PostgreSQL driver (nspostgres.so) is available from SourceForge. If you do decide to use nspostgres.so, you have to remember to change the AOLserver config file to point to nspostgres.so instead of postgres.so. This guide uses Mat Kovach's distro (i.e. postgres.so)
The patch that makes exec work on BSD is available at sourceforge.net
The patch that makes ns_uuencode work for binary files is available at sourceforge.net
The patch that makes AOLserver respect the -g flag is available at sourceforge.net

5: Re: API for parsing html-forms (response to 4)

Posted by Jeff Davis on 11/15/03 07:48 PM

openacs works just fine on 4.0 with no patching. The reason none of the docs mention using it is that the official release of 4.0 only happened 3 days ago.

The only change in install stuff is that you have to include the nsdb module and "package require tdom" rather than using nstdom (well besides building 4.0 which changed a lot as tcl has been broken out and aolserver is just linked against the standard tcl libs...).

Btw, where was that documentation from? We should probably update it.

6: Re: API for parsing html-forms (response to 1)

Posted by Nima Mazloumi on 11/17/03 10:03 AM

I copied that part from the cvs head-doc:

http://openacs.org/doc/openacs-HEAD/individual-programs.html

7: Re: API for parsing html-forms (response to 6)

Posted by Joel Aufrecht on 11/17/03 12:07 PM

I don't think we should switch the testing platform from AOLserver 3.3oacs1 to AOLserver 4 while in Beta on OpenACS 5.0. I think we should ship 5.0 and then consider AOLserver 4.0 support as a new feature in 5.1. By "support" I mean have some core developers AND automated test servers and installers using AOLserver 4 - we did do some testing against 4.0 and it uncovered a few subtle things, like changes in config.tcl, that we need to handle carefully.

8: Re: API for parsing html-forms (response to 7)

Posted by Jeff Davis on 11/17/03 01:18 PM

I put in a ticket for the out of date version stuff in software.xml.

As for 4.0, I have been using 4.0 since march for pretty much all the testing I have done and have not run into any problems (and I think Don, Bart, and Roberto have been running it too). It seems pretty bulletproof (although I agree its probably a 5.1 or 5.0.1 change).

9: Re: API for parsing html-forms (response to 1)

Posted by Nima Mazloumi on 11/17/03 01:30 PM

Is there any special step I have to consider before porting OpenACS to v4.0?

10: Re: API for parsing html-forms (response to 9)

Posted by Jeff Davis on 11/17/03 02:59 PM

You need tcl 8.4 (built with --enable-threads), you need to add nsdb.so to your modules, and you should use "package require tdom" to load tdom (rather than using nstdom).

Apart from the package require tdom, there are no changes needed to openacs.

11: Re: API for parsing html-forms (response to 7)

Posted by Andrew Piskorski on 11/18/03 10:32 PM

Joel, from a software stability assurance point of view your thoughts on sticking with AOLserver 3.3+patches for OpenACS 5.0 make sense, but in the long run we're all going to better off the faster we get everyone onto AOLserver 4.0. And a lot of OpenACS 5.0 users are, overall, going to find it easier to use AOLserver 4.0 than the older version, not harder.

So if people are willing to update the necessary docs and config files for in time, I'd OpenACS 5.0 should recommend and support both versions of AOLserver, with the note that in the future AOLserver 4.x will become the standard.

12: Re: API for parsing html-forms (response to 1)

Posted by Nima Mazloumi on 11/19/03 04:28 PM

Hi Jeff,
I installed TCL 8.4.4 and the AOLServer v4:

3 questions:

1. I compared the bin directory of Aolserver 3.3 and 4:

v3.3:
libphp4.so nscache.so nscp.so nsd-postgres nsfts.so nsopenssl.so nsperm.so nsrewrite.so nssock.so libtdom.so nscgi.so nsext.so nslog.so nspam.so nspostgres.so nssha1.so nsxml.so

v4:
nscgi.so nscp.so nsdb.so nsext.so nslog.so nsperm.so nssock.so

Can I simple copy these files or do I have to recompile them using the v4.0 source code.

2. Where do I have to make the setting package require tdom?

3. There is a sample-config.tcl file included in v4.0. Can I copy the missing settings from my current OpenACS config.tcl file? Do you have a config.tcl for OpenACS 5 file that the v4 server uses?

They still have the docs of v3 online at aolserver.org :(

Thank you for your help.

Greetings,
Nima

13: Re: API for parsing html-forms (response to 12)

Posted by Malte Sussdorff on 11/19/03 04:59 PM

Hi Nima,

though I'm not Jeff, I will answer anyway ;)

No you can't. Aolserver 3 and 4 work differently, just recompile them.
So far I have always linked it up and not used package require, but Jeff and I clearly have different opinions on this. In general though it seems that package require is the way openacs is going to suggest the use of AOLserver
Not as of now. Just make sure you include nsdb.so in your module section (ns_param nsdb ${bindir}/nsdb.so). And, depending on the way you install tdom, make sure this is included as well.
Use the -b yourip:yourport when starting AOLserver. Though only required for priviledged ports, it is useful to remember this and use it all the time.

14: Re: API for parsing html-forms (response to 1)

Posted by Nima Mazloumi on 11/19/03 05:58 PM

Hi Malte my friend. How are you?

Thank you for your reply. I thought I read somewhere that you wanted to write a little document on how to install the Aolserver 4. Am I right? Will it be on your companies web site?

Greetings,
Nima

15: Re: API for parsing html-forms (response to 14)

Posted by Malte Sussdorff on 11/19/03 06:16 PM

Actually I plan to put it into the CVS for 5.0. Furthermore it is already posted somewhere in the forums (though with a focus on "OS X". Not very detailed, just a step by step guide.

16: Re: API for parsing html-forms (response to 14)

Posted by Randy O'Meara on 11/20/03 01:48 AM

Malte graciously posted his instrcutions here:

http://openacs.org/forums/message-view?message_id=135723

17: Re: API for parsing html-forms (response to 1)

Posted by Nima Mazloumi on 11/20/03 11:40 AM

I found Maltes instructions under

http://openacs.org/forums/message-view?message_id=139390

Thank you Malte.