III.15.2 Webtest

API testing is only part of testing your package - it doesn't test the code in our adp/tcl pairs. For this, we can use TCLWebtest (see sourceforge).

TclWebtest is primarily for testing user interface and acceptance testing. It is a tool to write automated tests for web applications. It provides a simple API for issuing http requests, dealing with the result and assume specific response values, while taking care of the details such as redirects and cookies.
It has some basic html parsing functionality, to provide access to elements of the result html page that are needed for testing (mainly links and forms).

• TCLWebtest provides a library of functions (see command reference) that make it easy to call a page through HTTP, examine the results, and drive forms. TCLwebtest's functions overlap slightly with acs-automated-testing; see the example provided for one approach on integrating them.

• TCLWebtest tries to minimize the effort to write tests by implicitely assuming specific conditions whenever it makes sense. For example it always expects the server to return http codes other than 404 or 500, unless otherwise specified.

• The assertion procedures are targeted at test writers who want to make sure the behaviour of their web applications stays the same, without caring for style or minor wording changes. In the example below, it is just assumed that there is a link with the text "login" on the first page, that clicking on it results in a page with at least one form with at least two text-entry fields on it, and that submitting the form with the specified values results in a page that contains the "logged in" text.

• TCLWebtest should be suitable for testing larger chains of user interaction on a web application, for example a full ecommerce ordering session. tclwebtest could visit an ecommerce site as anonymous user, add some products to its shopping cart, check out the cart, register itself as user and enter a test address etc. The test script could also include the administration part of the interaction, by explicitely logging in as site admin, reviewing and processing the order, nuking the test user etc.

• TCLWebtest must be installed for to work. Since automated testing uses it, it should be part of every OpenACS installation. Note that TCLwebtest is installed automatically by Malte's install script.

Hint:

In order to simplify the generation of tclwebtest scripts the  Webtest-Recorder extension (TwtR) for Firefox is available see http://www.km.co.at/km/twtr This module is a plugin for Firefox. It is used to generate/edit a tclwebtest script which can be used later for regression testing without the need of a browser. There is a certain overlap of the application range between selenium and TwtR. This plugin was developed by Åsmund Realfsen for regression/load testing of the assessment module.

A typical script for tclwebtest looks like this:

set SERVER "testserver"
do_request "http://$SERVER/sometesturl/"
assert text "some text"

link follow "login"

field fill "testuser"
field fill "testpassword"
form submit

assert text "you are logged in as testuser"
    

 http://cvs.openacs.org/cvs/openacs-4/etc/install/tcl/twt-procs.tcl?rev=1.18

Command Reference:

• http://tclwebtest.sourceforge.net/doc/api_public.html
  

Here are some guidelines on how to write automated tests with TCLWebtest. It is a joy to work with automated testing once you get the hang of it. We will use the "myfirstpackage" as an example.

Create the directory that will contain the test script and edit the script file. The directory location and file name are standards which are recognized by the automated testing package:

[$OPENACS_SERVICE_NAME www]$ mkdir /var/lib/aolserver/$OPENACS_SERVICE_NAME/packages/myfirstpackage/tcl/test
[$OPENACS_SERVICE_NAME www]$ cd /var/lib/aolserver/$OPENACS_SERVICE_NAME/packages/myfirstpackage/tcl/test

[$OPENACS_SERVICE_NAME test]$ emacs myfirstpackages-procs.tcl

Write the tests. This is obviously the big step :) The script should first call ad_library like any normal -procs.tcl file:

ad_library {
    ...
}

To create a test case you call aa_register_case test_case_name.. Once you've created the test case you start writing the needed logic. We'll use the tutorial package, "myfirstpackage," as an example. Let's say you just wrote an API for adding and deleting notes in the notes packages and wanted to test that. You'd probably want to write a test that first creates a note, then verifies that it was inserted, then perhaps deletes it again, and finally verifies that it is gone.

Naturally this means you'll be adding a lot of bogus data to the database, which you're not really interested in having there. To avoid this I usually do two things. I always put all my test code inside a call to aa_run_with_teardown which basically means that all the inserts, deletes, and updates will be rolled back once the test has been executed. A very useful feature. Instead of inserting bogus data like: set name "Simon", I tend to generate a random script in order avoid inserting a value that's already in the database:

set name [ad_generate_random_string]

Here's how the test case looks so far:

aa_register_case mfp_basic_test {
    My test
} {
    aa_run_with_teardown  -rollback  -test_code  {

       }
}

Now look at the actual test code. That's the code that goes inside -test_code {}. We want to implement test case API-001, "Given an object id from API-001, invoke mfp::note::get. Proc should return the specific word in the title."

      set name [ad_generate_random_string]
      set new_id [mfp::note::add -title $name]
      aa_true "Note add succeeded" [exists_and_not_null new_id]

To test our simple case, we must load the test file into the system (just as with the /tcl file in the basic tutorial, since the file didn't exist when the system started, the system doesn't know about it.) To make this file take effect, go to the APM and choose "Reload changed" for "MyFirstPackage". Since we'll be changing it frequently, select "watch this file" on the next page. This will cause the system to check this file every time any page is requested, which is bad for production systems but convenient for developing. We can also add some aa_register_case flags to make it easier to run the test. The -procs flag, which indicates which procs are tested by this test case, makes it easier to find procs in your package that aren't tested at all. The -cats flag, setting categories, makes it easier to control which tests to run. The smoke test setting means that this is a basic test case that can and should be run any time you are doing any test. (a definition of "smoke test")

Once the file is loaded, go to ACS Automated Testing and click on myfirstpackage. You should see your test case. Run it and examine the results.

Example

Now we can add the rest of the API tests, including a test with deliberately bad data. The complete test looks like:

ad_library {
    Test cases for my first package.
}

    
aa_register_case  -cats {smoke api}  -procs {mfp::note::add mfp::note::get mfp::note::delete}  mfp_basic_test  {
A simple test that adds, retrieves, and deletes a record.
} {
aa_run_with_teardown  -rollback  -test_code {
set name [ad_generate_random_string]
set new_id [mfp::note::add -title $name]
aa_true "Note add succeeded" [exists_and_not_null new_id]
# Now check that the item exists
mfp::note::get -item_id $new_id -array note_array
aa_true "Note contains correct title" [string equal $note_array(title) $name]
# Now check, if titel got the value of name


   mfp::note::delete -item_id $new_id
          
                set get_again [catch {mfp::note::get -item_id $new_id -array note_array}]
                aa_false "After deleting a note, retrieving it fails" [expr $get_again == 0]
            }
    }
            
  

aa_register_case  -cats {api}  -procs {mfp::note::add mfp::note::get mfp::note::delete}  mfp_bad_data_test  {
        A simple test that adds, retrieves, and deletes a record, using some tricky data.
    } {
        aa_run_with_teardown  -rollback  -test_code  {
                set name {-Bad [BAD] \077 { $Bad}}    

                #Now name becomes this very unusual value: -Bad [BAD] \077 { $Bad}
                append name [ad_generate_random_string]
                set new_id [mfp::note::add -title $name]     

                #Now new_id becomes the value of the solution of proceduer add with starting argument $name as -title
                aa_true "Note add succeeded" [exists_and_not_null new_id]
                #Now test that new_id exists
                mfp::note::get -item_id $new_id -array note_array
                aa_true "Note contains correct title" [string equal $note_array(title) $name]
                aa_log "Title is $name"
                mfp::note::delete -item_id $new_id

                set get_again [catch {mfp::note::get -item_id $new_id -array note_array}]
                aa_false "After deleting a note, retrieving it fails" [expr $get_again == 0]
            }
    }