[omniORB] Solaris OmniOrb 4.0.3 server hanging and not responding
to incoming request
Michal Misiaszek
misiaszek at computaris.com
Thu Nov 11 16:29:58 GMT 2004
Hello,
We have serious problem with omniOrb. We are using version 4.0.3 on Sun
Solaris 8. We have a server which implements thread-per-connection
policy. In some unknown to us situation server stops processing request
from clients. The already created threads are processing and finalize
their task but the new incoming connection is accepted but new thread is
not spawned to serve it. Therefore client received TIMEOUT exception and
tries again without success. We have in general 8 connections in
parallel. For each request client opens new connection, sends request
and after receiving response it closes connection. In case of timeout
client opens (tries to) open another connection. Client is working in
connection-per-request thread in blocking mode. I have found similar
problem with number of allowed file descriptors on OmniOrb for Windows
ver 4.0.1 it was fixed in 4.0.3 (4).
http://www.omniorb-support.com/pipermail/omniorb-list/2004-March/025184.
html
Is there similar problem in Solaris ?
Here is an exception from OmniOrb log file:
omniORB: Distribution date: Fri Nov 21 17:39:25 GMT 2003 dgrisby
omniORB: Initialising omniDynamic library.
omniORB: Initialising incoming endpoints.
omniORB: Starting serving incoming endpoints.
omniORB: AsyncInvoker: thread id = 1 has started. Total threads = 1
omniORB: WARNING -- a nil (0) pointer is wrongly used as a
nil object reference.
omniORB: Adding root/my poa<smServer> (activating) to object table.
omniORB: Creating ref to local: root/my poa<smServer>
target id : IDL:omg.org/CORBA/Object:1.0
most derived id: IDL:SubscriptionManager:1.0
omniORB: Adding root/my poa<amServer> (activating) to object table.
omniORB: Creating ref to local: root/my poa<amServer>
target id : IDL:omg.org/CORBA/Object:1.0
most derived id: IDL:AccountManager:1.0
omniORB: Adding root/my poa<fafServer> (activating) to object table.
omniORB: Creating ref to local: root/my poa<fafServer>
target id : IDL:omg.org/CORBA/Object:1.0
most derived id: IDL:faf:1.0
omniORB: AsyncInvoker: thread id = 2 has started. Total threads = 3
omniORB: AsyncInvoker: thread id = 3 has started. Total threads = 3
omniORB: Accepted connection from giop:tcp:10.230.169.189:44580 because
of this rule: "* unix,ssl,tcp"
omniORB: AsyncInvoker: thread id = 4 has started. Total threads = 4
omniORB: Accepted connection from giop:tcp:10.230.169.189:44582 because
of this rule: "* unix,ssl,tcp"
omniORB: AsyncInvoker: thread id = 5 has started. Total threads = 5
omniORB: Accepted connection from giop:tcp:10.230.169.189:44583 because
of this rule: "* unix,ssl,tcp"
omniORB: AsyncInvoker: thread id = 6 has started. Total threads = 6
omniORB: AsyncInvoker: thread id = 7 has started. Total threads = 7
omniORB: Accepted connection from giop:tcp:10.230.169.189:44584 because
of this rule: "* unix,ssl,tcp"
omniORB: Handling a GIOP LOCATE_REQUEST.
omniORB: Handling a GIOP LOCATE_REQUEST.
omniORB: Handling a GIOP LOCATE_REQUEST.
omniORB: Handling a GIOP LOCATE_REQUEST.
omniORB: Handling a GIOP LOCATE_REQUEST.
omniORB: Handling a GIOP LOCATE_REQUEST.
omniORB: Handling a GIOP LOCATE_REQUEST.
omniORB: Handling a GIOP LOCATE_REQUEST.
omniORB: Handling a GIOP LOCATE_REQUEST.
omniORB: Handling a GIOP LOCATE_REQUEST.
...
omniORB: Handling a GIOP LOCATE_REQUEST.
omniORB: Handling a GIOP LOCATE_REQUEST.
omniORB: throw giopStream::CommFailure from
giopStream.cc:828(0,NO,COMM_FAILURE_UnMarshalArguments)
omniORB: Accepted connection from giop:tcp:10.230.169.189:34868 because
of this rule: "* unix,ssl,tcp"
omniORB: Handling a GIOP LOCATE_REQUEST.
omniORB: Handling a GIOP LOCATE_REQUEST.
omniORB: throw giopStream::CommFailure from
giopStream.cc:828(0,NO,COMM_FAILURE_UnMarshalArguments)
omniORB: Accepted connection from giop:tcp:10.230.169.189:34870 because
of this rule: "* unix,ssl,tcp"
omniORB: Handling a GIOP LOCATE_REQUEST.
omniORB: throw giopStream::CommFailure from
giopStream.cc:828(0,NO,COMM_FAILURE_UnMarshalArguments)
omniORB: Accepted connection from giop:tcp:10.230.169.189:34872 because
of this rule: "* unix,ssl,tcp"
omniORB: Handling a GIOP LOCATE_REQUEST.
omniORB: throw giopStream::CommFailure from
giopStream.cc:828(0,NO,COMM_FAILURE_UnMarshalArguments)
omniORB: Accepted connection from giop:tcp:10.230.169.189:34875 because
of this rule: "* unix,ssl,tcp"
omniORB: Handling a GIOP LOCATE_REQUEST.
omniORB: throw giopStream::CommFailure from
giopStream.cc:828(0,NO,COMM_FAILURE_UnMarshalArguments)
omniORB: Accepted connection from giop:tcp:10.230.169.189:34876 because
of this rule: "* unix,ssl,tcp"
omniORB: throw giopStream::CommFailure from
giopStream.cc:828(0,NO,COMM_FAILURE_UnMarshalArguments)
omniORB: Accepted connection from giop:tcp:10.230.169.189:34877 because
of this rule: "* unix,ssl,tcp"
omniORB: AsyncInvoker: thread id = 4929 has started. Total threads = 17
omniORB: Handling a GIOP LOCATE_REQUEST.
omniORB: throw giopStream::CommFailure from
giopStream.cc:828(0,NO,COMM_FAILURE_UnMarshalArguments)
omniORB: throw giopStream::CommFailure from
giopStream.cc:828(0,NO,COMM_FAILURE_UnMarshalArguments)
omniORB: Accepted connection from giop:tcp:10.230.169.189:34879 because
of this rule: "* unix,ssl,tcp"
omniORB: AsyncInvoker: thread id = 4930 has started. Total threads = 18
omniORB: Accepted connection from giop:tcp:10.230.169.189:34880 because
of this rule: "* unix,ssl,tcp"
omniORB: AsyncInvoker: thread id = 4931 has started. Total threads = 19
omniORB: throw giopStream::CommFailure from
giopStream.cc:828(0,NO,COMM_FAILURE_UnMarshalArguments)
omniORB: Accepted connection from giop:tcp:10.230.169.189:34885 because
of this rule: "* unix,ssl,tcp"
omniORB: AsyncInvoker: thread id = 4932 has started. Total threads = 20
omniORB: throw giopStream::CommFailure from
giopStream.cc:828(0,NO,COMM_FAILURE_UnMarshalArguments)
omniORB: Accepted connection from giop:tcp:10.230.169.189:34889 because
of this rule: "* unix,ssl,tcp"
omniORB: AsyncInvoker: thread id = 4933 has started. Total threads = 21
omniORB: throw giopStream::CommFailure from
giopStream.cc:828(0,NO,COMM_FAILURE_UnMarshalArguments)
omniORB: Accepted connection from giop:tcp:10.230.169.189:34890 because
of this rule: "* unix,ssl,tcp"
omniORB: AsyncInvoker: thread id = 4934 has started. Total threads = 22
omniORB: throw giopStream::CommFailure from
giopStream.cc:828(0,NO,COMM_FAILURE_UnMarshalArguments)
omniORB: Accepted connection from giop:tcp:10.230.169.189:34893 because
of this rule: "* unix,ssl,tcp"
omniORB: AsyncInvoker: thread id = 4935 has started. Total threads = 23
omniORB: throw giopStream::CommFailure from
giopStream.cc:828(0,NO,COMM_FAILURE_UnMarshalArguments)
omniORB: AsyncInvoker: thread id = 4936 has started. Total threads = 24
omniORB: Accepted connection from giop:tcp:10.230.169.189:34894 because
of this rule: "* unix,ssl,tcp"
omniORB: AsyncInvoker: thread id = 4937 has started. Total threads = 26
omniORB: throw giopStream::CommFailure from
giopStream.cc:828(0,NO,COMM_FAILURE_UnMarshalArguments)
omniORB: AsyncInvoker: thread id = 4938 has started. Total threads = 27
omniORB: Accepted connection from giop:tcp:10.230.169.189:34897 because
of this rule: "* unix,ssl,tcp"
omniORB: throw giopStream::CommFailure from
giopStream.cc:828(0,NO,COMM_FAILURE_UnMarshalArguments)
omniORB: AsyncInvoker: thread id = 4939 has started. Total threads = 27
omniORB: Accepted connection from giop:tcp:10.230.169.189:34898 because
of this rule: "* unix,ssl,tcp"
omniORB: AsyncInvoker: thread id = 4942 has started. Total threads = 30
omniORB: throw giopStream::CommFailure from
giopStream.cc:828(0,NO,COMM_FAILURE_UnMarshalArguments)
omniORB: throw giopStream::CommFailure from
giopStream.cc:828(0,NO,COMM_FAILURE_UnMarshalArguments)
omniORB: AsyncInvoker: thread id = 4940 has started. Total threads = 30
omniORB: Accepted connection from giop:tcp:10.230.169.189:34902 because
of this rule: "* unix,ssl,tcp"
omniORB: AsyncInvoker: thread id = 4941 has started. Total threads = 30
omniORB: Accepted connection from giop:tcp:10.230.169.189:34903 because
of this rule: "* unix,ssl,tcp"
omniORB: throw giopStream::CommFailure from
giopStream.cc:828(0,NO,COMM_FAILURE_UnMarshalArguments)
omniORB: Accepted connection from giop:tcp:10.230.169.189:34904 because
of this rule: "* unix,ssl,tcp"
omniORB: throw giopStream::CommFailure from
giopStream.cc:828(0,NO,COMM_FAILURE_UnMarshalArguments)
omniORB: Accepted connection from giop:tcp:10.230.169.189:34907 because
of this rule: "* unix,ssl,tcp"
<--- ... Here it hanged and was closed by kill
omniORB: AsyncInvoker: thread id = 4937 has exited. Total threads = 30
omniORB: AsyncInvoker: thread id = 4936 has exited. Total threads = 30
omniORB: AsyncInvoker: thread id = 4942 has exited. Total threads = 28
omniORB: AsyncInvoker: thread id = 4928 has exited. Total threads = 27
omniORB: AsyncInvoker: thread id = 4922 has exited. Total threads = 26
omniORB: AsyncInvoker: thread id = 4929 has exited. Total threads = 26
omniORB: AsyncInvoker: thread id = 4931 has exited. Total threads = 24
omniORB: AsyncInvoker: thread id = 4900 has exited. Total threads = 24
omniORB: AsyncInvoker: thread id = 4924 has exited. Total threads = 24
omniORB: AsyncInvoker: thread id = 4925 has exited. Total threads = 21
omniORB: AsyncInvoker: thread id = 4923 has exited. Total threads = 20
omniORB: AsyncInvoker: thread id = 4930 has exited. Total threads = 19
omniORB: AsyncInvoker: thread id = 4910 has exited. Total threads = 18
omniORB: AsyncInvoker: thread id = 4921 has exited. Total threads = 17
omniORB: AsyncInvoker: thread id = 4912 has exited. Total threads = 16
omniORB: AsyncInvoker: thread id = 4939 has exited. Total threads = 16
omniORB: AsyncInvoker: thread id = 4918 has exited. Total threads = 16
omniORB: AsyncInvoker: thread id = 4932 has exited. Total threads = 16
omniORB: AsyncInvoker: thread id = 4926 has exited. Total threads = 12
omniORB: AsyncInvoker: thread id = 4933 has exited. Total threads = 12
omniORB: AsyncInvoker: thread id = 4935 has exited. Total threads = 10
omniORB: AsyncInvoker: thread id = 4927 has exited. Total threads = 10
omniORB: AsyncInvoker: thread id = 4920 has exited. Total threads = 8
omniORB: AsyncInvoker: thread id = 4940 has exited. Total threads = 8
omniORB: AsyncInvoker: thread id = 4938 has exited. Total threads = 8
omniORB: AsyncInvoker: thread id = 4941 has exited. Total threads = 5
omniORB: AsyncInvoker: thread id = 4894 has exited. Total threads = 4
omniORB: AsyncInvoker: thread id = 4934 has exited. Total threads = 3
omniORB: Preparing to shutdown ORB.
omniORB: Destroying POA(RootPOA).
omniORB: Destroying POA(my poa).
omniORB: Deactivating all POA(my poa)'s objects.
omniORB: Waiting for requests to complete on POA(my poa).
omniORB: Requests on POA(my poa) completed.
omniORB: Etherealising POA(my poa)'s objects.
omniORB: Removing root/my poa<fafServer> (etherealising) from object
table
omniORB: Removing root/my poa<amServer> (etherealising) from object
table
omniORB: Removing root/my poa<smServer> (etherealising) from object
table
omniORB: Destruction of POA(my poa) complete.
omniORB: Deactivating all POA(RootPOA)'s objects.
omniORB: Waiting for requests to complete on POA(RootPOA).
omniORB: Requests on POA(RootPOA) completed.
omniORB: Etherealising POA(RootPOA)'s objects.
omniORB: Stopping serving incoming endpoints.
And pstack when it was hanging:
----------------- lwp# 2 / thread# 2 -------------------- fde9ed78
signotifywait () fe74e950 _dynamiclwps (fe76c000, 0, ff3e7aac, 1000,
ff3d4124, ff3e8008) + 1c fe751c2c thr_yield (0, 0, 0, 0, 0, 0) + 8c
----------------- lwp# 5 / thread# 5 -------------------- fe7591fc
lwp_cond_wait (fe76cd28, fe76cd10, fd909d30) fe748998 _co_timerset
(fe76cd28, fe76c000, fe76cd10, fd909d30, fe76c000, 0) + a0 fe75b01c
_thread_start (0, 0, 0, 0, 0, 0) + 40
----------------- lwp# 57 / thread# 9 -------------------- fde9f428
lwp_sema_wait (fd501e60)
fe7496dc _park (fd501e60, fe76c000, 0, fd501d98, 2316c, fd807d98) +
114
fe7493a4 _swtch (fd501d98, 0, fe76c000, 5, 1000, 0) + 424
fe747d88 cond_reltimedwait (0, fd501d98, 0, fe76c000, 0, 0) + 1dc
fe747b9c cond_timedwait (1480008, 147ffe0, fd501af0, 1480008, 0, 0) + 2c
fe747b04 pthread_cond_timedwait (1480008, 147ffe0, fd501af0, 0, 0,
fd501af0) + c 00115790 timedwait__14omni_conditionUlUl (1480000,
419108aa, 3937c970, 0, 0, 22f6c4) + 18 001f6124
execute__Q24omni9Scavenger (1466110, 2da400, 2da400, fd501b68, fd501b6c,
0) + a0 001b658c run__19omniAsyncWorkerInfo (14cb5c8, 3250c8, 0,
fffffff8, 0, 147b7bd) + 1e4 002731b8 run__15omniAsyncWorkerPv (14cb5c8,
0, 273198, 4, 14cb5c8, 0) + 20 00115d74 omni_thread_wrapper (14cb5c8,
fdff5d38, 0, 5, 1, fe401000) + a4 fe75b01c _thread_start (14cb5c8, 0,
0, 0, 0, 0) + 40
----------------- lwp# 40 / thread# 7 --------------------
fde9d5b8 _poll (74, 0, 0, fd7055f8, fdebf1bc, fd705590) + 8
fe75aa28 select (74, fd705a30, 0, 0, fd705ab0, 73) + 34
002363e0 Select__Q24omni16SocketCollection (14b9e18, 0, 1, 0, 0, 0) +
168 00230ce8
AcceptAndMonitor__Q24omni11tcpEndpointPFPvPQ24omni14giopConnection_vPv
(14b9e10, 1fc308, 14ba390, 230c4c , 329048, 14b9e10) + 9c 001fc3cc
execute__Q24omni16giopRendezvouser (14ba390, 326e78, 1fc324, 14ba390, 0,
0) + a8 001b658c run__19omniAsyncWorkerInfo (149d290, 3250c8, 0,
fffffff8, 0, 1480079) + 1e4 002731b8 run__15omniAsyncWorkerPv (149d290,
0, 273198, 4, 149d290, 0) + 20 00115d74 omni_thread_wrapper (149d290,
fdfe3d38, 0, 5, 1, fe401000) + a4 fe75b01c _thread_start (149d290, 0,
0, 0, 0, 0) + 40
----------------- lwp# 91 / thread# 6 -------------------- fde9f428
lwp_sema_wait (fd807e60)
fe7496dc _park (fd807e60, fe76c000, 0, fd807d98, 2316c, fd501d98) +
114
fe7493a4 _swtch (fd807d98, 0, fe76c000, 5, 1000, 0) + 424
fe747d88 cond_reltimedwait (0, fd807d98, 0, fe76c000, 0, 0) + 1dc
fe747b9c cond_timedwait (fd807c78, fd807c60, fd807c88, fd807c78, 19fdc,
fde97654) + 2c fe747b04 pthread_cond_timedwait (fd807c78, fd807c60,
fd807c88, ff324a90, 0, fd807c90) + c ff003338 SGPsleep (0, 64, 258, 1,
ff006938, fe99a850) + 2a8 ff003524 SgpRefreshThread (0, fdfe3d38, 0, 5,
1, fe401000) + 108 fe75b01c _thread_start (0, 0, 0, 0, 0, 0) + 40
----------------- lwp# 73 / thread# 3 -------------------- fde9f428
lwp_sema_wait (fdb0de60)
fe7496dc _park (fdb0de60, fe76c000, 0, fdb0dd98, 2316c, fd807d98) +
114
fe7493a4 _swtch (fdb0dd98, fdb0dd98, fe76c000, 5, 1000, 1) + 424
fe74d9b8 _reap_wait (fe770980, fe76c000, 0, 134d, 0, 168) + 38 fe74d710
_reaper (fe76ce00, fe772708, fe770980, fe76cdd8, 1, fe400000)
+ 38 fe75b01c _thread_start (0, 0, 0, 0, 0, 0) + 40
----------------- lwp# 97 --------------------------------
fde9c91c door (0, 0, 0, 0, fdca5d38, 4)
fe7567c8 _sc_door_func (0, 0, 0, 0, 0, 0) + 54
----------------- lwp# 84 -------------------------------- fe7591fc
lwp_cond_wait (fe7734e8, fe7734f8, fe76cd80)
fe748cc4 _age (3e, fe76cd6c, fe76c000, fe778d04, 0, fe401000) + 74
fe748c48 _qswtch (f9509d98, fd3d5d38, 1, fe778d04, 0, 2) + 118
-------------------------- thread# 1 -------------------- fe7481ac
cond_wait (32f458, 0, 0, fe76c000, 0, 0) + 11c fe748070
pthread_cond_wait (32bc08, 32bbd0, fe76c000, 1, 32f500, 0) + 8 00115730
wait__14omni_condition (32bc00, 2d9800, 2d9800, 0, 81010100, 0) + 8
001a352c perform__Q24omni15ORBAsyncInvokerUlUl (1490bc0, 0, 0, 323940,
1a348c, 1490bc0) + a0 001a25d0 run__10omniOrbORB (13f1798, 13f1798,
1a251c, 7, 294d60, 1cf6f70) + b4 000c34bc CORBA_SAM_run__FiPPc
(ffbef1a0, 3257b0, 14ba2b8, 325780, 14ba2b8, 325828) + ef4
000c57d8 main (2, ffbef394, ffbef3a0, 32d074, 0, 0) + 19b8
000ae558 _start (0, 0, 0, 0, 0, 0) + 5c
-------------------------- thread# 4 -------------------- fe747d88
cond_reltimedwait (0, fda0bd98, 0, fe76c000, 0, 0) + 1dc fe747b9c
cond_timedwait (fda0b7e8, fda0b7d0, fda0b7f8, fda0b7e8, 298460, 0) + 2c
fe747b04 pthread_cond_timedwait (fda0b7e8, fda0b7d0, fda0b7f8, 0,
298450, fda0b800) + c 000e27b8 licSleep (15270, 549c, 15270, 0, 298450,
298450) + 248 000dd98c CheckLicenceThread (0, fdfe3d38, 1, fe778d04, 0,
2) + 1f0 fe75b01c _thread_start (0, 0, 0, 0, 0, 0) + 40
-------------------------- thread# 8 -------------------- fe747d88
cond_reltimedwait (0, fd603d98, 0, fe76c000, 0, 0) + 1dc fe747b9c
cond_timedwait (fd603bf0, fd603bd8, fd603c00, fd603bf0, fe9d0634, 9) +
2c fe747b04 pthread_cond_timedwait (fd603bf0, fd603bd8, fd603c00, 9,
fe9ae068, fd603c08) + c fe8fb958 PPBsleep (258, 2eb6, fe9b3510, 8,
fe9b63b8, fe9b6360) + 7cc fe98484c periodicallyRefreshParameters
(fe9b6460, fe9b6460, fd603d38, fd603d10, 0, 0) + 274 fe984d14
periodicallyRefreshParametersThread (0, fdff5d38, 0, 5, 1, fe401000) +
58 fe75b01c _thread_start (0, 0, 0, 0, 0, 0) + 40
(END)
I will very appriciate prompt response as it is very critical problem to
me. Looking forward to your answer. Best regards Michal Misiaszek
More information about the omniORB-list
mailing list