Forum OpenACS Q&A: http requests with XoTCL

Collapse
Posted by Raúl Morales Hidalgo on
Hi there,

As I'm completely new to xotcl, I'd like to know a couple of things:

- can I have a similar approach to what ns_httppost does in tcl (i.e. make a post to a url and get the contents) with xotcl ::xo::HttpRequest ?

-Is it possible to make requests with https to a non standard port?

I would gladly apreciate any example that shows me how to do a request this way, I'm having issues with http.tcl module and this is as good reason as any other to start using xotcl.

thanks in advance.

Collapse
Posted by Stefan Sobernig on
Hi Raúl ...

As for 1), you got the right pointer: ::xo::HttpRequest
Take the following as example:

A sample POST request:

set request [::xo::HttpRequest new \
-content_type "text/plain" \
-url $url \
-post_data "your payload" \
-request_header_fields [list _fieldname_ _fieldvalue_]]

$r set data

The last line will give you the result. If you leave out post_data (or pass an empty string), the requests turns into a GET request (simply encode the parameters into the string passed to "-url"). The call "$r set statusCode" will give you the returned HTTP status code. Connection problems (socket-level etc.) will show up as an empty data variable on the request object, i.e. $r.

As for 2), there is currently no stock tls/ssl option in xotcl-core. But it shouldn't be that hard to add it, e.g. using the tls package. I think I got a working example somewhere that I created quite a time ago. I will post it as soon as I find it. Maybe Gustaf Neumann has an ace up his sleeve?!

Collapse
Posted by Raúl Morales Hidalgo on
Hi Stefan,

Thanks a lot, it almost worked ;)

I understand $r in your last line is $request (at least that worked for me :P)

I have no problems with GET requests, but when I want to do a POST request I have no clear idea of what to do, add a string à la GET in post_data field (-post_data "user=$user&comm_id=$comm_id) or add it in the headers as a POSTDATA header?

Thanks in advance,

Raúl

Collapse
Posted by Stefan Sobernig on
Sorry, I had the idea to rewrite it and got stuck at the beginning. so the complete example should read as:
set request [::xo::HttpRequest new \
-content_type "text/plain" \
-url $url \
-post_data "your payload" \
-request_header_fields [list _fieldname_ _fieldvalue_]]
$request set data

The above example yields a POST request, simply by adding a non-empty value to "-post_data".

A GET request would look like the following:

set url "http://localhost:8000/xosoap/services/";
set request [::xo::HttpRequest new \
-content_type "text/html" \
-url [export_vars -base $url {{s badge}}]]
$request set data

The subtle difference is setting post_data or not! Whether you format the post or query string data in a "parameter'ish" way, does not make a difference.

Collapse
Posted by Tom Jackson on
Wow, that is pretty non-intutitive. How do you do a HEAD or PUT?
Collapse
Posted by Gustaf Neumann on
Raúl

there is now an updated version of the HTTP client classes in cvs head, which has two improvements

 1) support for https (via the tcl module TLS)
 2) some documentation (see below).

Additional HTTP methods will be added when needed.

Hope this helps

Best regards
-gustaf neumann


XOTcl implementation for synchronous and asynchronous HTTP and HTTPs requests

  #
  # Defined classes
  #  1) HttpRequest
  #  2) AsyncHttpRequest
  #  3) HttpRequestTrace (mixin class)
  #  4) Tls (mixin class, applicable to various protocols)
  #
  ######################
  #
  # 1 HttpRequest
  #
  # HttpRequest is a class to implement the client side
  # for the HTTP methods GET and POST.
  #
  # Example of a GET request:
  #
  #  set r [::xo::HttpRequest new -url http://www.openacs.org/]
  #
  # The resulting object $r contains all information
  # about the requests, such as e.g. status_code or 
  # data (the response body from the server). For details
  # look into the output of [$r serialize]. The result 
  # object in $r is automatically deleted at cleanup of
  # a connection thread.
  #
  # Example of a POST request with a form with var1 and var2
  # (providing post_data causes the POST request).
  #    
  #  set r [::xo::HttpRequest new \
  #             -url http://yourhost.yourdomain/yourpath \
  #             -post_data [export_vars {var1 var2}] \
  #             -content_type application/x-www-form-urlencoded \ 
  #         ]
  #
  # Provided that the Tcl module tls (see e.g. http://tls.sourceforge.net/)
  # is available and can be loaded via "package require tls" into 
  # the aolserver, you can use both TLS/SSL secured or unsecured requests 
  # in the synchronous/ asynchronous mode by using an
  # https url.
  # 
  #  set r [::xo::HttpRequest new -url https://learn.wu-wien.ac.at/]
  #
  ######################
  #
  # 2 AsyncHttpRequest
  #
  # AsyncHttpRequest is a subclass for HttpRequest implementing
  # asynchronous HTTP requests without vwait (vwait causes 
  # stalls on aolserver). AsyncHttpRequest requires to provide a listener 
  # or callback object that will be notified upon success or failure of 
  # the request.
  #
  # Asynchronous requests are much more complex to handle, since
  # an application (a connection thread) can submit multiple
  # asynchronous requests in parallel, which are likely to
  # finish after the current request is done. The advantages
  # are that the spooling of data can be delegated to a spooling
  # thead and the connection thread is available for handling more
  # incoming connections. The disadvantage is the higher
  # complexity, one needs means to collect the received data.
  #
  # The following example uses the background delivery thread for
  # spooling and defines in this thread a listener object (a). 
  # Then in the second step, the listener object is used in te
  # asynchronous request (b).
  #
  # (a) Create a listener/callback object in the background. Provide
  # the two needed methods, one being invoked upon success (deliver),
  # the other upon failure or cancellation (done).
  #
  #  ::bgdelivery do Object ::listener \
  #     -proc deliver {payload obj} {
  #       my log "Asynchronous request suceeded!"
  #     } -proc done {reason obj} {
  #       my log "Asynchronous request failed: $reason"
  #     }
  #
  # (b)  Create the actual asynchronous request object in the background. 
  # Make sure that you specify the previously created listener/callback 
  # object as "request_manager" to the request object.
  #
  #  ::bgdelivery do ::xo::AsyncHttpRequest new \
  #     -url "https://oacs-dotlrn-conf2007.wu-wien.ac.at/conf2007/" \
  #     -request_manager ::listener
  #
  ######################
  #
  # 3 HttpRequestTrace
  #
  # HttpRequestTrace can be used to trace the one or all requests.
  # If activated, the class writes protocol data into 
  # /tmp/req-<somenumber>.
  #
  # Use 
  #
  #  ::xo::HttpRequest instmixin add ::xo::HttpRequestTrace
  #
  # to activate trace for all requests, 
  # or mixin the class into a single request to trace it.
  #
Collapse
Posted by Tom Jackson on

There is an undocumented API in AOLserver 4.5 [ns_http]. Below is an example of use, although a poor one. I don't think that the code in AOLserver is complete, but I'll describe how it works.

set timeout 20 
set method GET
set body ""
set headerSet [ns_set create]

set id [ns_http queue \
           -timeout $timeout\
           -method $method \
           -body $body \
           -headers $headerSet]

set re [ns_http wait \
            -elapsed elapsedVar \
            -result resultVar \
            -headers headersSetVar \
            -status statusVar \
            $id]

There are a few other commands (cancel and cleanup), but how to use this? Connection threads can queue requests and another worker thread or threads can pull from the list and service them. For instance, nsv_lappend could be used to post the ids and another set of worker threads could pull from the list. If the 'wait' is not successful, the task is not removed from the queue, so it could be canceled, or tried again later. Another nsv could track the number of tries. Also the worker thread could cancel and rewrite the request to extend wait timeout. The worker thread/s could also just be a scheduled proc. The proc could even reschedule itself so you never have two running at once, or you could chain scheduled procs. One master could loop over the list and distribute the load in some way.

But unless you use multiple worker threads this isn't async, except that you can pass off the work to something other than a connection thread, and you can process more than one request per thread, but they are handled in series.

set url http://192.168.1.102:8001/runproc
for {set i 0} {$i < 10} {incr i} {
    if {$i} {
	set action queue
    } else {
	set action run
    }
    set result$i [ns_http $action -timeout 200 http://192.168.1.102:8001/runproc]
    ns_log Notice "queued [set result$i]"
}

for {set i 0} {$i < 10} {incr i} {
    ns_log Notice "waiting for [set result$i]"
    set data$i [ns_http wait [set result$i]]
    ns_log Notice "finished waiting for [set result$i]"
}

for {set i 0} {$i < 10} {incr i} {
    append allData "data$i ([set result$i] = '[set data$i]'\n"
}

ns_return 200 text/plain $allData 
Collapse
Posted by Raúl Morales Hidalgo on
Gustaf, Tom

Thanks a lot to both of you, I haven't had time to test it but it's much more than enough to solve my issues. I'll post again here once I've been able to test.

Again, thanks

Collapse
Posted by Malte Sussdorff on
Ehm.... I just struggled for a couple of hours figuring out why my jodconverter service was complaining for an invalid output mime/type. After digging through their java code I realized that the Accept Header was set to */*. So I thought I set it with request_header_fields in ::xo:HttpRequest. Still no luck. Until I found that send_request sets this automatically to */*......

To get this to work nicely I made a change like this, which allows me to call the jodconverter with Accept: application/pdf to tell it that I want to have a PDF document returned to me. Byte the way, this makes the whole code for OpenOffice document PDF generation even easier than it was before. And I thought I could make a consulting business out of it.... Bad luck ......

Index: tcl/http-client-procs.tcl
===================================================================
--- tcl/http-client-procs.tcl (Revision 475)
+++ tcl/http-client-procs.tcl (Arbeitskopie)
@@ -144,6 +144,7 @@
Attribute port
Attribute path -default "/"
Attribute url
+ Attribute accept -default "*/*"
Attribute post_data -default ""
Attribute content_type -default "text/plain"
Attribute request_header_fields -default {}
@@ -302,11 +303,11 @@
}

HttpCore instproc send_request {} {
- my instvar S post_data host
+ my instvar S post_data host accept
if {[catch {
set method [expr {$post_data eq "" ? "GET" : "POST"}]
puts $S "$method [my path] HTTP/1.0"
- puts $S "Accept: */*"
+ puts $S "Accept: $accept"
puts $S "Host: $host"
puts $S "User-Agent: [my user_agent]"
foreach {tag value} [my request_header_fields] {

Collapse
Posted by Gustaf Neumann on
Malte,

good catch. I have actually no idea, why "accept: */*" was there (the rfc says, it is optional). It is certainly not a good idea to hard-code the transmission.

Since i see no good reason to set this header-field differently than other request header fields, i took another approach and removed the automatic transmission of "accept" altogether. If a client wants to specify "accept", it can be done - like for all other header fields - via request_header_fields, seems that it was your first guess as well.

-gustaf neumann
PS: fixed in CVS head.