Forum OpenACS Q&A: Tracking Memory Usage

Collapse
Posted by Dave Bauer on
Hi,

Are there useful techniques for tracking memory usage? One of my sites crashes regularly with an unable to alloc message with various amounts of memory sometimes up to 512mb.

I know one problem with growing memory usage/leak is Tcl exec and I have been using the ns_proxy exec wrapper for a while now. I also went in and found a few places where tDom documents were not freed after use. Are there any other common mistakes I have missed?

Collapse
2: Re: Tracking Memory Usage (response to 1)
Posted by Benjamin Brink on
What OS?

Sometimes db caching can consume significant amounts of a system with limited memory.

Collapse
Posted by Dave Bauer on
Thanks.

I have 6G or RAM free with plenty of DB caching.

I am specifically referring the to size of the NSD process.

Collapse
4: Re: Tracking Memory Usage (response to 1)
Posted by Gustaf Neumann on
The list of potential leaks is quite long and depends on many factors. if possible, tell us more about the site.

several things to check:
- use a recent version of tcl 8.5
- are you running a 32bit or 64bit binary?
- do you make daily restarts?
- what tcl extensions are you using in which version?
- what openacs packages are you using?
- what is the rss and vmsize of nsd after startup and after one hour?
- what is the maxconnections in your aolserver configuration?
- have you set some especially large configuration values?

domNodes are a classical memleak, but one can as easy leak in namespaced variables, since these are not reclaimed at the end of a requests.

tcl is optimized for speed and not for memory space. try to avoid cases, where you read e.g. huge files into tcl vars and append to it. If the allocated memory is not sufficient (e.g. by one byte), tcl will realloc the memory with the double size.

aolserver has a few small memory leaks, naviserver is better in this regard. Compare the memory growth of openacs.org [1], which runs aolserver, with the size from next-scripting.org [2] which runs naviserver. If the traffic on your site is similar to openacs.org and you run a similar configuration, then the memory growth rate should be as well comparable to openacs.org.

One can do a lot with different memory allocators, but i am not sure this is necessary.

-gustaf

[1] https://openacs.org/munin/localdomain/localhost.localdomain/naviserver_openacs_memsize.html
[2] http://alice.wu-wien.ac.at:8000/munin/wu.ac.at/alice.wu.ac.at/naviserver_next_memsize.html
[3] https://next-scripting.org/xowiki/docs/misc/thread-mallocs/index1

Collapse
Posted by Dave Bauer on
I found a daily procedure that was loading into a tcl variable and parsing a 256mb XML file.

I disabled that as it was a legacy function no longer needed.

I will have to rewrite that as a plain tcl cron job as it was not really using any OpenACS or AOLserver functions that can't be replaced by regular tcl.

So that explains the unable to alloc 512mb notice at least. Size is down to around 500mb stable for 24 gours so far.

Collapse
Posted by Dave Bauer on
So assuming I can restart in around 1 minute I can reenable the nightly restart since I won't have a significant downtime with that process. That should keep it overall stable.
Collapse
Posted by Gustaf Neumann on
... parsing a 256MB file ...

Note that if one has a Tcl-obj of size X, and appends to it, Tcl tends to double the allocation size to omit further copying operations, when one is again appending to that string. This strategy is quite runtime efficient when more appends are happening later, but as well quite wasteful when X is e.g. 1 GB. A similar situation might happen depending on the Tcl version and Tcl operation, when Tcl converts the string into UCS-2, where every character is represented by two bytes. Also, when parsing the XML file, the parser will need space for its data structures and copies of the string chunk.

So, altogether i would not be surprised, if the parsing would use 1 GB of memory. The bad thing about the zippy allocator is that it does not give memory back to the operating system, so the nsd process will grow by such operations, but it will not shrink later. Using different allocators can help here, but these require tcl modifications.

Another option is to use ns_proxy [1], which is run as a separate process. When this process is terminated all memory is returned back to the OS. It might be less effort for you to move the parsing to the ns_proxy instead of a separate cronjob.

-gustaf neumann

PS: a similar bulky memory growth might happen in the filters within search, when e.g. huge pdf files or the like are indexed.

[1] http://naviserver.sourceforge.net/n/nsproxy/files/doc/mann/ns_proxy.html