Forum OpenACS Q&A: AOLserver & OpenACS Benchmarks

I have been getting meager performance from my OpenACS sites. I ran 
ApacheBench against both an AOLserver-only site and an out-of-the-box 
OpenACS site, using the default AOLserver page. 

Summary: AOLserver/3.2+ad10 (1176.47 rps), OpenACS 3.2.4 (6.81 rps)

How can I improve this? Based on my server, shouldn't I be getting 
closer to 20-30 rps on an out-of-the-box OpenACS site?:

Both sites are running on the same server, separate IPs (no virtual 
hosting), and logging was turned off during the tests:

PIII 500 MHz, 320 MB memory, RAID1 IDE (8ms access time), RedHat 6.2 

 10:27pm  up 57 days, 12:33,  2 users,  load average: 0.09, 0.05, 0.09
97 processes: 96 sleeping, 1 running, 0 zombie, 0 stopped
CPU states:  0.4% user,  0.1% system,  0.3% nice,  0.5% idle
Mem:   322628K av,  160148K used,  162480K free,  160788K shrd,    
8900K buff
Swap:  530064K av,   88560K used,  441504K free                   
40108K cached

-------------------------------------- 
AOLserver/3.2+ad10: 

[nsadmin@roam apache]$ bin/ab -n 700 -c 50 
http://roam.unifiedmind.com:8000/index.html
This is ApacheBench, Version 1.3c <$Revision: 1.45 $> apache-1.3
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, 
http://www.zeustech.net/
Copyright (c) 1998-2000 The Apache Group, http://www.apache.org/

Server Software:        
AOLserver/3.2+ad10                                 
Server Hostname:        roam.unifiedmind.com
Server Port:            8000

Document Path:          /index.html
Document Length:        1689 bytes

Concurrency Level:      50
Time taken for tests:   0.595 seconds
Complete requests:      700
Failed requests:        0
Total transferred:      1366040 bytes
HTML transferred:       1199190 bytes
Requests per second:    1176.47
Transfer rate:          2295.87 kb/s received

Connnection Times (ms)
              min   avg   max
Connect:        1     7    14
Processing:    18    32    41
Total:         19    39    55

------------------------------------------------ 
OpenACS 3.2.4: 

[nsadmin@roam apache]$ bin/ab -n700 -c50 
http://electricspeed.com/aolserver.html
This is ApacheBench, Version 1.3c <$Revision: 1.45 $> apache-1.3
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, 
http://www.zeustech.net/
Copyright (c) 1998-2000 The Apache Group, http://www.apache.org/

Benchmarking electricspeed.com (be patient)...
Server 
Software:                                                           
Server Hostname:        electricspeed.com
Server Port:            80

Document Path:          /aolserver.html
Document Length:        1691 bytes

Concurrency Level:      50
Time taken for tests:   102.831 seconds
Complete requests:      700
Failed requests:        0
Total transferred:      1363757 bytes
HTML transferred:       1195411 bytes
Requests per second:    6.81
Transfer rate:          13.26 kb/s received

Connnection Times (ms)
              min   avg   max
Connect:        0     3    57
Processing:   823  7085 22441
Total:        823  7088 22498

Thanks for the reply. All modules have been disabled except for nssock, and I simply copied the default AOLserver page to the www directory of an uncustomized OpenACS 3.2.4 install -- it's a static page (see http://electricspeed.com/aolserver.html).

However, the nsd.tcl file for this server is configured so that files having the .html extension are parsed as ADP pages. BUT, I get similar results after giving aolserver.html a different extension so that it isn't parsed as an ADP page (i.e. aolserver.shtml):

[nsadmin@roam aolserver]$ /usr/local/apache/bin/ab -n 700 -c 50 http://electricspeed.com/aolserver.shtml
This is ApacheBench, Version 1.3c <$Revision: 1.45 $> apache-1.3
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 1998-2000 The Apache Group, http://www.apache.org/

Server Software:        AOLserver/3.2+ad10                                 
Server Hostname:        electricspeed.com
Server Port:            80

Document Path:          /aolserver.shtml
Document Length:        1689 bytes

Concurrency Level:      50
Time taken for tests:   110.872 seconds
Complete requests:      700
Failed requests:        0
Total transferred:      1473500 bytes
HTML transferred:       1182300 bytes
Requests per second:    6.31
Transfer rate:          13.29 kb/s received

Connnection Times (ms)
              min   avg   max
Connect:        0     3    34
Processing:  1656  7669 28536
Total:       1656  7672 28570

What is probably happening is that performance sucks because there is the overhead of running all the startup tcl whenver a new thread is initialized.  Since you are initializing 50 threads, there is a noticeable delay in starting up all those interpreters - most likely your nsd.tcl file does not have MaxThreads=50 or anything like that.

Alternatively, what is happening is that you are limited in the number of threads the server is running; in this case, there are say 20 threads that are handling multiple high-speed connections.

I would say that having 50 simultaneous connections all of which do not use keepalive is probably not very real world.  Further, you are only running 700 connections total - so the thread startup stuff is skewing the results.  You can check this by running with -n 7000 and seeing what happens - the avg number in the summary should drop.

That being said, I have seen similar numbers during my testing.  My guess is that the filters that determine whether you are able to access the page slows things down.  Yet it does not seem sluggish when my clients hit the pages, even over high speed DSL or cable modems.

As a side issue, it would be interesting to see how much load the 50 simultaneous Apache's are placing on your system.  The best tool to view this is usually "vmstat 5" - look for the context switch and CPU idle numbers.

Here are results from testing with 7000 connections with keepalive OFF (its corresponding vmstat results are posted to http://jamesthornton.com/acs/vmstat.txt):

(Results for 700 and 7000 connections with keepalive ON follow.)

[nsadmin@roam aol32]$ /usr/local/apache/bin/ab -n 7000 -c 50 http://electricspeed.com/aolserver.shtml
This is ApacheBench, Version 1.3c <$Revision: 1.45 $> apache-1.3
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 1998-2000 The Apache Group, http://www.apache.org/

Server Software:        AOLserver/3.2+ad10                                 
Server Hostname:        electricspeed.com
Server Port:            80

Document Path:          /aolserver.shtml
Document Length:        1689 bytes

Concurrency Level:      50
Time taken for tests:   978.370 seconds
Complete requests:      7000
Failed requests:        0
Total transferred:      14735000 bytes
HTML transferred:       11823000 bytes
Requests per second:    7.15
Transfer rate:          15.06 kb/s received

Connnection Times (ms)
              min   avg   max
Connect:        0     1    64
Processing:   930  6961 29135
Total:        930  6962 29199

------------------------------------
700 connections keapalive on:

[nsadmin@roam aol32]$ /usr/local/apache/bin/ab -k -n 700 -c 50 http://electricspeed.com/aolserver.shtml
This is ApacheBench, Version 1.3c <$Revision: 1.45 $> apache-1.3
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 1998-2000 The Apache Group, http://www.apache.org/

Server Software:        AOLserver/3.2+ad10                                 
Server Hostname:        electricspeed.com
Server Port:            80

Document Path:          /aolserver.shtml
Document Length:        1689 bytes

Concurrency Level:      50
Time taken for tests:   111.357 seconds
Complete requests:      700
Failed requests:        0
Keep-Alive requests:    700
Total transferred:      1477000 bytes
HTML transferred:       1182300 bytes
Requests per second:    6.29
Transfer rate:          13.26 kb/s received

Connnection Times (ms)
              min   avg   max
Connect:        0     1    26
Processing:  1586  7709 32402
Total:       1586  7710 32428

-------------
7000 connections keepalive on:

[nsadmin@roam aol32]$ /usr/local/apache/bin/ab -k -n 7000 -c 50 http://electricspeed.com/aolserver.shtml
This is ApacheBench, Version 1.3c <$Revision: 1.45 $> apache-1.3
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 1998-2000 The Apache Group, http://www.apache.org/

Server Software:        AOLserver/3.2+ad10                                 
Server Hostname:        electricspeed.com
Server Port:            80

Document Path:          /aolserver.shtml
Document Length:        1689 bytes

Concurrency Level:      50
Time taken for tests:   963.632 seconds
Complete requests:      7000
Failed requests:        0
Keep-Alive requests:    7000
Total transferred:      14770000 bytes
HTML transferred:       11823000 bytes
Requests per second:    7.26
Transfer rate:          15.33 kb/s received

Connnection Times (ms)
              min   avg   max
Connect:        0     0    37
Processing:   947  6861 25473
Total:        947  6861 25510

As part of the work I'm doing for a client, I benchmarked the performance of OpenACS 4 and some code that I have developed for this client. I used aD's Traffic Jamme to do the tests. I also ran ApacheBench, but the results are similar.

The webserver and DB server (PostgreSQL) sit on separate machines. The DB server is a Gateway dual PIII 933 server with 1GB of memory and running RAID 10. The webserver is a dual PIII 500Mhz box with 1GB of memory. Both boxes run FreeBSD.

I tested several different pages in the system which had database queries. The queries ranged from a simple permissions check to a query on a tree. Traffic Jamme reported 4 connections/sec for the most complex page (the tree query with some TCL processing) and 17 connections/sec for a simple page with a single query into the acs_permissions table. I ran Traffic Jamme with 50 threads/20 iterations.

I ran top on the DB server and the webserver and found out that the DB server is only using 10% of the CPUs and the webserver is only using 50% of the CPUs. I tried playing around with the MaxThreads=MinThreads=X number to see if I could increase the CPU usage. I tried turning off logging completely. Nothing seemed to increase the maximum number of connections/sec.

Is there something else I can try to do to use more of the CPU time on either server? Is there a bottleneck that I am overlooking? Thanks!

I compiled the linuxthreads port for FreeBSD and recompiled AOLServer to use linuxthreads.  linuxthreads causes each thread to be treated as a process.  After restarting AOLServer, I ran top and watched the threads spread to both CPUs during the benchmark.  It looks like FreeBSD can put individual processes on separate CPUs, which was confirmed when I looked at the postmaster running on the other FreeBSD machine.  Each connection was a separate process and FreeBSD spread the load across the CPUs.

As a result of the tweaking I doubled the connection/sec since I started benchmarking on this particular page (3 connections/sec to 7 connections/sec).  PostgreSQL is not getting worked at all so I am convinced that another webserver will help greatly.  For this server (dual PIII, 1GB RAM, OpenACS 4, PG 7.1.2), it looks like the maximum connections/sec is 17 (just serving the default aolserver index.html page).

I'm compiling a FreeBSD install/tuning guide which I will send to Roberto as soon as I finish.

For this server (dual PIII, 1GB RAM, OpenACS 4, PG 7.1.2), it looks like the maximum connections/sec is 17 (just serving the default aolserver index.html page).

Stephen - The server can sustain 45-50 requests per second during the test, but it drops off for some reason and the average is 17 req/sec.  Maybe it's an issue with Traffic Jamme.  I wanted to try it with ApacheBench, but I couldn't get it to hit the IP address.  It kept crashing.

For a simple db page (ie permissions check), I got 17 connections/sec.  There might be something wrong with my benchmarking technique.  For the static HTML page, I left all of the necessary aolserver modules on.

Gilbert

Both CPUs are pegged on the webserver.

Gilbert

Best Regards,

For instance you could disable the permission on specific path as described in http://www.arsdigita.com/bboard/q-and-a-fetch-msg?msg%5fid=000cfi&topic%5fid=21&topic=web%2fdb