[omniORB] giopServer::deactivate hangs

Wed Jun 25 10:44:20 BST 2003

I'm using omniORB 4.0.0 on Solaris 8, and have a problem that sounds just
like this one

http://www.omniorb-support.com/pipermail/omniorb-list/2003-May/023581.html

I didn't see that it was resolved, so I'll post the details I have.  I have
processes communicating via omniORB like so

  A ----> B <----> C

B spawns C (fork/exec) and passes an IOR on the command line to it.  C
sends it's IOR to B via a pipe set up by B prior to the fork.  I'm having
trouble getting C to shut down consistently.  If I have B sequentially
spawn and signal 50 instances of C, perhaps 2 of them will hang.  I'm using
what I hope is the most robust means of shutting down a CORBA server
process:
      - Each CORBA server has a signal handler for SIGTERM.  The handler
simply sets a global flag and signals a condition variable.
      - prior to calling ORB::run(), the main thread spawns a thread which
waits on the condition variable.  When signalled this thread calls
ORB::shutdown(0).  This thread also has SIGTERM masked so it shouldn't be
the one to call the signal handler.
      - The main thread will clean up my background thread when ORB::run()
returns.

For every other server this method of signalling shutdown has been
infalible.  For this fork/exec'd server it hangs now and then and if I have
to kill it then B will not shutdown nicely either.  I'm pretty sure B has
released all object references to C prior to signalling it, but I would
hope that wouldn't matter.  Shouldn't an ORB unconditionally return from
run() if there are no active invocations at the time?

Running with traceLevel 25 I see normal shutdowns look like this ( and all
happens very quickly ):

omniORB: (?) ObjRef(IDL:SSPCommon/EventConsumer:1.0) -- deleted.
omniORB: (?) Preparing to shutdown ORB.
omniORB: (?) Starting an ORB shutdown thread.
omniORB: (4) ORB shutdown thread started.
omniORB: (4) Destroying POA(RootPOA).
omniORB: (4) Deactivating all POA(RootPOA)'s objects.
omniORB: (4) State root<0> (active) -> deactivating
omniORB: (4) Waiting for requests to complete on POA(RootPOA).
omniORB: (4) Requests on POA(RootPOA) completed.
omniORB: (4) State root<0> (deactivating) -> etherealising
omniORB: (4) Etherealising POA(RootPOA)'s objects.
omniORB: (4) Removing root<0> (etherealising) from object table
omniORB: (4) Object table entry root<0> (dead) deleted.
omniORB: (4) RefCountServantBase has zero ref count -- deleted.
omniORB: (4) Stopping serving incoming endpoints.
omniORB: (4) giopServer waits for completion of rendezvousers and workers
omniORB: (5) AsyncInvoker: thread id = 5 has started. Total threads = 4
omniORB: (5) giopWorker task execute.
omniORB: (5) throw giopStream::CommFailure from
giopStream.cc:812(0,NO,COMM_FAIL
URE_UnMarshalArguments)
omniORB: (5) Server connection refcount = 1
omniORB: (4) giopServer back from waiting.
omniORB: (5) Server connection refcount = 0
omniORB: (5) Server close connection from giop:tcp:161.222.1.105:52523
omniORB: (4) Destruction of POA(RootPOA) complete.
omniORB: (4) Shutting-down all incoming endpoints.
omniORB: (4) TCP endpoint shut down.
omniORB: (4) 0 object references present at ORB shutdown.
omniORB: (4) ORB shutdown is complete.
omniORB: (0) Deinitialising omniDynamic library.
omniORB: (0) Client connection refcount (forced) = 0
omniORB: (0) Client close connection to giop:tcp:161.222.1.105:51360
omniORB: (0) 0 remaining bidir ropes deleted.
omniORB: (0) 1 remaining rope deleted.
omniORB: (2) AsyncInvoker: thread id = 2 has exited. Total threads = 4
omniORB: (1) AsyncInvoker: thread id = 1 has exited. Total threads = 4
omniORB: (3) AsyncInvoker: thread id = 3 has exited. Total threads = 2
omniORB: (5) AsyncInvoker: thread id = 5 has exited. Total threads = 1
omniORB: (0) AsyncInvoker: deleted.
omniORB: (0) No more references to the ORB -- deleted.
omniORB: (0) Final clean-up
omniORB: (0) Released 340 static TypeCodes.
omniORB: (0) Deleted 4 nil object references and 2 other tracked objects.
omniORB: (0) Final clean-up completed.

An abnormal shutdown looks like this:

omniORB: (?) ObjRef(IDL:SSPCommon/EventConsumer:1.0) -- deleted.
omniORB: (?) Preparing to shutdown ORB.
omniORB: (?) Starting an ORB shutdown thread.
omniORB: (4) ORB shutdown thread started.
omniORB: (4) Destroying POA(RootPOA).
omniORB: (4) Deactivating all POA(RootPOA)'s objects.
omniORB: (4) State root<0> (active) -> deactivating
omniORB: (4) Waiting for requests to complete on POA(RootPOA).
omniORB: (4) Requests on POA(RootPOA) completed.
omniORB: (4) State root<0> (deactivating) -> etherealising
omniORB: (4) Etherealising POA(RootPOA)'s objects.
omniORB: (4) Removing root<0> (etherealising) from object table
omniORB: (4) Object table entry root<0> (dead) deleted.
omniORB: (4) RefCountServantBase has zero ref count -- deleted.
omniORB: (4) Stopping serving incoming endpoints.
omniORB: (4) giopServer waits for completion of rendezvousers and workers
omniORB: (5) AsyncInvoker: thread id = 5 has started. Total threads = 4
omniORB: (5) giopWorker task execute.
omniORB: (5) throw giopStream::CommFailure from
giopStream.cc:812(0,NO,COMM_FAIL
URE_UnMarshalArguments)
omniORB: (5) Server connection refcount = 1

then it hangs a while and eventually I see these two additional lines

omniORB: (2) Client connection refcount (forced) = 0
omniORB: (2) Client close connection to giop:tcp:161.222.1.105:57244

then it hangs again.  After many more minutes I get these lines (btw, why
is the Total threads not dropping if threads are exiting?)

omniORB: (1) AsyncInvoker: thread id = 1 has exited. Total threads = 4
omniORB: (5) AsyncInvoker: thread id = 5 has exited. Total threads = 4
omniORB: (3) AsyncInvoker: thread id = 3 has exited. Total threads = 4

then it hangs again.

Using pstack, I see the state of the process is

18949:  ../bin/lpserver_iomg -slave
IOR:000000000000002049444c3a535350436f6d6d
-----------------  lwp# 1  --------------------------------
 feaf9770 lwp_cond_wait (feb15548, feb15558, feb0edb0)
 feae90ac _age     (0, feb0ed9c, feb0e000, feb1ad8c, 0, fe401000) + 74
 feae9030 _qswtch  (fd80fd70, feb14740, 0, 5, 1, fe401000) + 118
-----------------  lwp# 2 / thread# 2  --------------------
 fea9e9a4 signotifywait ()
 feaeed54 _dynamiclwps (feb0e000, ff3d1610, fef909dc, ff3e66b4, fe8615c7,
0) + 1c
 feaf2030 thr_yield (0, 0, 0, 0, 0, 0) + 8c
-----------------  lwp# 3  --------------------------------
 feaf9770 lwp_cond_wait (feb15548, feb15558, feb0edb0)
 feae90ac _age     (fe9b5d70, feb0ed9c, feb0e000, feb1ad8c, 0, fe401000) +
74
 feae9030 _qswtch  (fdf05d70, fe9b5d10, 0, 5, 1, fe401000) + 118
-----------------  lwp# 4 / thread# 6  --------------------
 fea9f054 lwp_sema_wait (fe007e30)
 feae9ac4 _park    (fe007e30, feb0e000, 0, fe007d70, 24d84, 0) + 114
 feae978c _swtch   (fe007d70, 0, feb0e000, 5, 1000, 0) + 424
 feae7e10 cond_reltimedwait (0, 0, 0, 1, 0, 0) + 1f8
 feae7c08 cond_timedwait (447480, 447458, fe007ad0, 447480, 0, 0) + 2c
 feae7b70 pthread_cond_timedwait (447480, 447458, fe007ad0, 0, feff43b4, 0)
+ c
 fefe1fb8 __1cOomni_conditionJtimedwait6MLL_i_ (447478, 3ef9b466, 2c803078,
0, 0, 2c803078) + 40
 ff0cfa80 __1cEomniJScavengerHexecute6M_v_ (451768, 2894, 2800, 0,
ff165b80, 0) + cc
 ff087d58 __1cPomniAsyncWorkerIreal_run6M_v_ (452af0, 0, 1, 45f038, 451768,
ff16fd4c) + 21c
 ff087f40 __1cPomniAsyncWorkerDrun6Mpv_v_ (452af0, 0, feb0e000, 0, 452af0,
0) + 40
 fefe2658 omni_thread_wrapper (452af0, feff3f94, 1, feb1ad8c, 0, 2) + d4
 feafb728 _thread_start (452af0, 0, 0, 0, 0, 0) + 40
-----------------  lwp# 5  --------------------------------
 feaf9770 lwp_cond_wait (feb15548, feb15558, feb0edb0)
 feae90ac _age     (0, feb0ed9c, feb0e000, 0, 0, 4) + 74
 fea9c5a4 _door_return (fe7e5cb0, feaea740, 0, 0, 0, 0) + 68
-----------------  lwp# 6 / thread# 8  --------------------
 fea9f054 lwp_sema_wait (feb0fa08)
 feae8d04 _co_timerset (feb0ed30, feb0e000, 1, 3, feb0e000, 0) + f4
 feafb728 _thread_start (0, 0, 0, 0, 0, 0) + 40
-----------------  lwp# 7  --------------------------------
 fea9c548 door     (0, 0, 0, 0, fe7d5d10, 4)
 feaf6ba4 _sc_door_func (0, 0, 0, 0, 0, 0) + 54
--------------------------  thread# 1  --------------------
 feae826c cond_wait (4390a8, 0, 0, feb0e000, 0, ff16c4a8) + 13c
 feae8110 pthread_cond_wait (ff16c4e8, ff16c4a8, feb0e000, 1, feb69978, 8)
+ 8
 fefe1f1c __1cOomni_conditionEwait6M_v_ (ff16c4e0, 0, 0, 25c, 2800, 29c0) +
18
 ff06aff0 __1cEomniPORBAsyncInvokerHperform6MLL_v_ (ff16c4a8, 0, 0, 0,
feb69978, 8) + 94
 0004b070 __1cLunsafe_main6Fippc_i_ (3, ffbeed74, 13640, 0, feb69978, 4) +
c18
 0004b328 main     (9, ffbeed74, ffbeed9c, 41a800, 0, 0) + 110
 0004a370 _start   (0, 0, 0, 0, 0, 0) + 108
--------------------------  thread# 3  --------------------
 feaeddbc _reap_wait (feb129e0, feb0e000, 0, 7, 0, 7) + 38
 feaedb14 _reaper  (feb0ee30, feb14740, feb129e0, feb0ee08, 1, fe400000) +
38
 feafb728 _thread_start (0, 0, 0, 0, 0, 0) + 40
--------------------------  thread# 5  --------------------
 feae826c cond_wait (fe109d70, 0, 0, feb0e000, 0, 44f5b0) + 13c
 feae8110 pthread_cond_wait (44f6d8, 44f5b0, ff2fcb4c, 0, 0, 0) + 8
 febdc2fc __1cHdtgutilNcDTGconditionEwait6M_v_ (4516e4, 1, 1, 0, 0, 0) + 4c
 fec09e84 __1cHdtgutilIcHandlerDrun6M_v_ (44ff70, 0, 0, 0, 0, 0) + 94
 fec183ac cPThread_start_func (44ff70, fe7f5d10, 0, 5, 1, fe401000) + 44
 feafb728 _thread_start (44ff70, 0, 0, 0, 0, 0) + 40
--------------------------  thread# 9  --------------------
 feae826c cond_wait (fdd01d70, 0, 0, feb0e000, 0, 45f298) + 13c
 feae8110 pthread_cond_wait (45f2b8, 45f298, feabc000, 0, feabfc90,
fdd01810) + 8
 fefe1f1c __1cOomni_conditionEwait6M_v_ (45f2b0, 0, ff110e9c, 0, 0, 4507b8)
+ 18
 ff0d5150 __1cEomniKgiopServerKdeactivate6M_v_ (45f278, 44f730, 450950,
44f730, ff16e364, 0) + 2a8
 ff0d4878 __1cEomniKgiopServerEstop6M_v_ (45f278, bfc, 2800, 29c0,
ff16e364, 19) + 28
 ff0918c0 __1cEomniOomniObjAdapterPadapterInactive6M_v_ (451364, feb0e000,
800, bd4, 1, 0) + bc
 ff0ab034 __1cEomniKomniOrbPOAKdo_destroy6Mb_v_ (451358, fdd01ac4, 1,
451364, ff13f7e1, 0) + 338
 ff0a604c __1cEomniKomniOrbPOAHdestroy6Mbb_v_ (451358, 2c00, 1, 2400, 253c,
1) + 23c
 ff0afb74 __1cEomniKomniOrbPOAIshutdown6F_v_ (1004, ff16e7f8, feabc000, 0,
fb0e8, feaebb44) + 6c
 ff06aacc __1cKomniOrbORBPactual_shutdown6M_v_ (44fef8, feb0e000, fb020,
ff16c4a8, 0, 250) + 44
 ff06abd0 __1cSshutdown_thread_fn6Fpv_v_ (44fef8, 2, ff06ab5c, ff06ab5c,
460c78, 0) + 74
 fefe2604 omni_thread_wrapper (460c78, feff3f94, 0, 5, 1, fe401000) + 80
 feafb728 _thread_start (460c78, 0, 0, 0, 0, 0) + 40

Perhaps there's something more I can do prior to exec()?  Do I need to
close all unwanted file descriptors inherited from the parent process?

Any help is appreciated.

Kendall