Forum OpenACS Development: Re: Image resizing, exec and server problems

25: Re: Image resizing, exec and server problems (response to 24)

Posted by Malte Sussdorff on 05/06/08 08:21 PM

Hi Stefan, I know this might ask for a lot but you seem to have your head around this pretty tightly. Could this trick be put into a procedure that uses the XOTcl bgdelivery feature so we could call something like "xo_exec" ? Or, alternatively, could you implement your idea on something non-core which uses exec (e.g. photo album) so it can be tested and maybe adopted?

Russel, to answer your question: I usually get to around 50 - 60 calls to exec, before I run into an out of memory problem calling the exec, which then results into a restart of the server every I would say 120 calls. But this is rough estimation and really depends on your memory.

26: Re: Image resizing, exec and server problems (response to 25)

Posted by Stefan Sobernig on 05/07/08 10:33 AM

Malte

you seem to have your head around this pretty tightly.

I'd wish, in the meantime I learnt that the pipe indirection through open also involves a fork(). But, still, async i/o + bgdelivery make sense and provide agility. The tricky thing is to avoid background zombie processes.

I will try to come up with a little solution using bgdelivery that can be tested.

27: Re: Image resizing, exec and server problems (response to 25)

Posted by russell muetzelfeldt on 05/08/08 07:45 AM

hmmm... that number scared me, but I've just run a quick test with 20 parallel clients each making 100 requests, each of which required 20 [exec] calls to service (so 40,000 exec calls in ~10 min spread across 10 connection threads)... every single one completed successfully, all my threads are still there and accepting connections, and the nsd process hasn't grown by any significant amount... perhaps the TCL threaded fork issue has been fixed? (debian etch, kernel 2.6.22-4-amd64, tcl 8.4.12-1.1, aolserver4 4.0.10-7)

28: Re: Image resizing, exec and server problems (response to 27)

Posted by Stefan Sobernig on 05/08/08 09:41 AM

Russell!

As long as it works for you and you tested (with a minimum effort) that it scales (to whatever extent) than it is apparently fine (at least).

However, another critical thing is the kind of exec call itself (redirector, background, ...) and the code surrounding it. Malte never posted the critical exec line, at least to my knowledge.

In other words, if it works for you. Go ahead!

My option referred more to use a non-blocking variant and link it to the background delivery feature (a larger picture of design).

I agree while there are resources on comp.lang.tcl etc. that report on the threaded fork() issue, I cannot find a concise statement on the issue's state (fixed, open, unsolvable).

29: Re: Image resizing, exec and server problems (response to 28)

Posted by russell muetzelfeldt on 05/08/08 12:03 PM

as I understand it the surrounding TCL code is irrelevant, as is whether the fork is the result of calling [exec], [open "|whatever"] or anything else... the problem is TCL doesn't/didn't implement pthread_atfork, so threads might lock up any time a new process is fork()/exec()'d by TCL... I understand what you're talking about with running the event loop to avoid blocking, but we don't have any problem with long-running children blocking IO for other threads so it's not related to the problem I thought I might be facing... looks interesting though...

30: Re: Image resizing, exec and server problems (response to 29)

Posted by Gustaf Neumann on 05/08/08 02:32 PM

russell, have you considered using tclmagic?
it allows image resizing, there are no forks involved, since it can be loaded into aolserver

Short intro: http://wiki.tcl.tk/9775