[omniORB] omniORB 4.0.5 server hangs under heavy load

[p.klotz at icoserve.com]

Hi

We encounter sporadic hangs of our omniORB 4.0.5 based servers. The 
servers operate in thread pool mode with 5 threads under Red Hat 
Enterprise Linux 3 Update 3.

In a torture test 40 clients connect to the server in parallel. Each 
client performs a single method call on the server (a simple "void ping()" 
method), exits and is started again. In this scenario it takes from a few 
minutes to a few hours until the server stops responding. It does not 
matter if the clients and the server run on the same or on different 
machines.

Here is a trace output at level 10 (Messages starting with a '+' were 
added by me in order to better understand what is going on):

omniORB: (2) +omniAsyncInvoker::insert: omniTask::AnyTime: pd_idle_threads
omniORB: (6) throw giopStream::CommFailure from 
giopStream.cc:831(0,NO,COMM_FAILURE_UnMarshalArguments)
omniORB: (2) +omniAsyncInvoker::insert: omniTask::AnyTime, 
omniTask*=0x9db8ea8, pd_idle_threads=0x9db87e0, pd_nthreads=5, 
pd_maxthreads=5, pd_totalthreads=6
omniORB: (2) +omniAsyncInvoker::insert: omniTask::AnyTime: pd_idle_threads
omniORB: (6) throw giopStream::CommFailure from 
giopStream.cc:831(0,NO,COMM_FAILURE_UnMarshalArguments)
omniORB: (2) +omniAsyncInvoker::insert: omniTask::AnyTime, 
omniTask*=0x9db8af0, pd_idle_threads=0x9db87e0, pd_nthreads=5, 
pd_maxthreads=5, pd_totalthreads=6
omniORB: (2) +omniAsyncInvoker::insert: omniTask::AnyTime: pd_idle_threads
omniORB: (2) +omniAsyncInvoker::insert: omniTask::AnyTime, 
omniTask*=0x9db8b18, pd_idle_threads=(nil), pd_nthreads=5, 
pd_maxthreads=5, pd_totalthreads=6
omniORB: (6) throw giopStream::CommFailure from 
giopStream.cc:831(0,NO,COMM_FAILURE_UnMarshalArguments)
omniORB: (2) +omniAsyncInvoker::insert: omniTask::AnyTime: else
omniORB: (3) throw giopStream::CommFailure from 
giopStream.cc:831(0,NO,COMM_FAILURE_UnMarshalArguments)
omniORB: (4) throw giopStream::CommFailure from 
giopStream.cc:831(0,NO,COMM_FAILURE_UnMarshalArguments)
omniORB: (3) AsyncInvoker: thread id = 3 has exited. Total threads = 6
omniORB: (4) AsyncInvoker: thread id = 4 has exited. Total threads = 6
omniORB: (6) AsyncInvoker: thread id = 6 has exited. Total threads = 6
omniORB: (5) AsyncInvoker: thread id = 5 has exited. Total threads = 3

At some point in time all worker threads exit and although clients want to 
connect, no new worker threads are started. The server produces 100% CPU 
load and all clients hang.

So far I was not able to produce hangs at trace levels higher than 10. The 
additional output seems to lower the probability of the hangs.

In
http://www.omniorb-support.com/pipermail/omniorb-list/2004-November/026184.html
Duncan provides a patch for a problem that I would consider at least very 
similar to this one. I applied the patch to omniORB 4.0.5 but the 
resulting server blocks without starting a single worker thread.

Any tips and hints are welcome.

Best regards, Peter.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.omniorb-support.com/pipermail/omniorb-list/attachments/20050116/305c8b79/attachment.htm