[omniORB] crash in openssl code

Fri Feb 23 12:10:47 UTC 2024

Hi again,

unfortunately this is an omniORB problem with bidirectional and ssl. By using a mutex around all calls using pd_ssl

in sslConnection.cc I found out that Recv snd Send are then used at the same time by different threads. Recv is used

sometimes in blocking mode so simply using a mutex is not the solution as omniorb will than hang completely.
The only option for us to use ssl now is to switch bidirectional CORBA off.
It's just strange that this does not happen more often...

Greetings,
    Michael

On 2/21/24 14:13, Michael Teske via omniORB-list wrote:
>
> Hi,
>
>
> I've created an issue
>
> https://github.com/openssl/openssl/issues/23650
>
> but now the question arose if omniORB does ensure to use a ssl connection only from one thread. I looks like it's so, but maybe there's a bug somewhere?
>
>
> Regards,
>
>   Michael
>
>
> On 2/21/24 10:40, Michael Teske via omniORB-list wrote:
>>
>> Hi,
>>
>> as required by our customers, we switched to use omniorb with ssl. This works fine so far,
>>
>> but on our machine with openssl3 (running RHEL9), we get crashes from time to time looking like this:
>>
>> (gdb) where
>> #0  0x00007f26bb0e14df in tls_get_message_header (mt=<synthetic pointer>, s=0x7f2684001940) at ssl/statem/statem_lib.c:1167
>> #1  read_state_machine (s=0x7f2684001940) at ssl/statem/statem.c:587
>> #2  state_machine (s=0x7f2684001940, server=<optimized out>) at ssl/statem/statem.c:442
>> #3  0x00007f26bb0d2969 in ssl3_read_bytes (s=<optimized out>, type=23, recvd_type=0x0, buf=0x7f25b0000d48 "", len=8192, peek=0, readbytes=0x7f25e69f39f0) at 
>> ssl/record/rec_layer_s3.c:1711
>> #4  0x00007f26bb0ab7fc in ssl3_read_internal (s=0x7f2684001940, buf=0x7f25b0000d48, len=8192, peek=0, readbytes=0x7f25e69f39f0) at ssl/s3_lib.c:4462
>> #5  0x00007f26bb0b2137 in SSL_read (s=<optimized out>, buf=<optimized out>, num=<optimized out>) at ssl/ssl_lib.c:1885
>> #6  0x00007f26bbfbd1bf in omni::sslConnection::Recv (this=0x7f2684016b78, buf=0x7f25b0000d48, sz=<optimized out>, deadline=...)
>>     at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/ssl/sslConnection.cc:246
>> #7  0x00007f26bbf04b88 in omni::giopStream::inputMessage (this=this at entry=0x7f25b0000b68) at 
>> /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/giopStream.cc:841
>> #8  0x00007f26bbf175a3 in omni::giopImpl12::inputNewServerMessage (g=0x7f25b0000b68) at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/giopImpl12.cc:438
>> #9  0x00007f26bbf184a8 in omni::giopImpl12::inputMessageBegin (g=0x7f25b0000b68, unmarshalHeader=<optimized out>)
>>     at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/giopImpl12.cc:646
>> #10 0x00007f26bbf0bc81 in omni::GIOP_S::dispatcher (this=0x7f25b0000b60) at /home/michael.teske/build/thirdparty/trader/omniORB/include/omniORB4/internal/giopStream.h:143
>> #11 0x00007f26bbf096b5 in omni::giopWorker::execute (this=0x7f2684017db0) at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/giopWorker.cc:77
>> #12 0x00007f26bbec38d0 in omniAsyncWorker::real_run (this=0x7f268401a490) at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/invoker.cc:578
>> #13 0x00007f26bbec487f in omniAsyncPoolServer::workerRun (this=<optimized out>, worker=<optimized out>)
>>     at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/invoker.cc:328
>> #14 0x00007f26bbec3499 in omniAsyncWorker::mid_run (this=0x7f268401a490) at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/invoker.cc:511
>> #15 0x00007f26bbec47de in omniAsyncWorker::run (this=0x7f268401a490) at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/invoker.cc:126
>> #16 0x00007f26bbfcd1d5 in omni_thread_wrapper (ptr=0x7f268401a490) at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omnithread/posix.cc:459
>> #17 0x00007f26baa9f802 in start_thread (arg=<optimized out>) at pthread_create.c:443
>> #18 0x00007f26baa3f450 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
>>
>> I did dnf debuginfo-install openssl-libs-3.0.7-24.el9.x86_64 and am thus able to look into openssl source. It looks as if it gets wrong data in ssl3_read_bytes
>>
>> and crashes then because s->init_buf is NULL, probably already resetted in tls_finish_handshake().
>>
>> In the source code around ssl/record/rec_layer_s3.c:1711 it says:
>>
>> /*
>>
>> * Unexpected handshake message (ClientHello, NewSessionTicket (TLS1.3) or
>>
>> * protocol violation)
>>
>> */
>>
>> if((s->rlayer.handshake_fragment_len>= 4)
>>
>> && !ossl_statem_get_in_handshake(s)) {
>>
>> intined = (s->early_data_state== SSL_EARLY_DATA_READING);
>>
>> /* We found handshake data, so we're going back into init*/
>>
>> ossl_statem_set_in_init(s, 1);
>>
>> i = s->handshake_func(s);
>>
>> (s->handshake_func is ossl_statem_connect which calls state_machine(s,0)) . While this looks like an openssl bug to me (it should call SSL_Fatal or re-create the buffer),
>>
>> the question remains, why does this happen?
>>
>> The thread causing this is here:
>>
>> (gdb) where
>> #0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7f26a9a24730, op=393, expected=0, futex_word=0x7f26840016d0) at futex-internal.c:57
>> #1  __futex_abstimed_wait_common (futex_word=futex_word at entry=0x7f26840016d0, expected=expected at entry=0, clockid=clockid at entry=0, abstime=abstime at entry=0x7f26a9a24730,
>>     private=private at entry=0, cancel=cancel at entry=true) at futex-internal.c:87
>> #2  0x00007f26baa9c3ff in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word at entry=0x7f26840016d0, expected=expected at entry=0, clockid=clockid at entry=0,
>>     abstime=abstime at entry=0x7f26a9a24730, private=private at entry=0) at futex-internal.c:139
>> #3  0x00007f26baa9eea4 in __pthread_cond_wait_common (abstime=0x7f26a9a24730, clockid=0, mutex=0x8349e0, cond=0x7f26840016a8) at pthread_cond_wait.c:504
>> #4  ___pthread_cond_timedwait64 (cond=0x7f26840016a8, mutex=0x8349e0, abstime=0x7f26a9a24730) at pthread_cond_wait.c:644
>> #5  0x00007f26bbfcc9a4 in omni_condition::timedwait (this=0x7f26840016a0, secs=<optimized out>, nanosecs=<optimized out>)
>>     at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omnithread/posix.cc:175
>> #6  0x00007f26bbf03ec5 in omni_condition::timedwait (t=..., this=<optimized out>) at /home/michael.teske/build/thirdparty/trader/omniORB/include/omnithread.h:363
>> #7  omni::giopStream::sleepOnRdLockAlways (this=0x7f2684001748) at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/giopStream.cc:392
>> #8  0x00007f26bbf17334 in omni::giopImpl12::inputReplyBegin (g=0x7f2684001748, unmarshalHeader=0x7f26bbf1aa90 <omni::giopImpl12::unmarshalReplyHeader(omni::giopStream*)>)
>>     at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/giopImpl12.cc:557
>> #9  0x00007f26bbf0a85c in omni::GIOP_C::ReceiveReply (this=0x7f2684001740) at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/GIOP_C.cc:126
>> #10 0x00007f26bbeef2a6 in omniRemoteIdentity::dispatch (this=0x7f2684000d70, call_desc=...) at /home/michael.teske/build/thirdparty/trader/omniORB/include/omniORB4/IOP_C.h:137
>> #11 0x00007f26bbed83ec in omniObjRef::_invoke (this=0x7f2684001208, call_desc=..., do_assert=false) at 
>> /home/michael.teske/build/thirdparty/trader/omniORB/include/omniORB4/omniObjRef.h:202
>> #12 0x00007f26bbed8593 in omniObjRef::_remote_is_a (this=<optimized out>, a_repoId=<optimized out>)
>>     at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/omniObjRef.cc:343
>> #13 0x00007f26bbed87e3 in omniObjRef::_realNarrow (this=0x7f2684001208, repoId=0x6d88c0 "IDL:exchangeAgent/eaServer:1.0")
>>     at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/omniObjRef.cc:174
>> #14 0x00000000005eae9b in exchangeAgent::eaServer::_narrow (obj=<optimized out>) at libidl/EAGeneral_skel.cpp:845
>> #15 0x0000000000526687 in LS::narrow_with_timeout<exchangeAgent::eaServer> (timeout=5000, x=0x7f2684001260) at 
>> /home/build/Builds/Trader-build1605/Trader/src/LoginServer/EAInfo.cpp:112
>> #16 LS::UsedEA::run (this=0x89e120) at /home/build/Builds/Trader-build1605/Trader/src/LoginServer/EAInfo.cpp:576
>> #17 0x00000000006ca225 in omniJTCThread::entrance_hook (this=0x89e330) at /home/build/Builds/Trader-build1605/Trader/src/libomniJTC/Thread.cpp:1037
>> #18 0x00007f26bbfcd14e in omni_thread_wrapper (ptr=0x89e7e0) at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omnithread/posix.cc:449
>> #19 0x00007f26baa9f802 in start_thread (arg=<optimized out>) at pthread_create.c:443
>> #20 0x00007f26baa3f450 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
>>
>>
>> The object reference it tries to narrow is from an url like this
>>
>> corbaloc:ssliop:rhel9build:33344/ExchangeAgent
>>
>>
>> The process providing this url was in the middle of starting up. It creates its main server object using a POA with
>>
>> omniPolicy::PLAIN_OBJECT_KEYS_ENABLE and BiDirPolicy::BIDIRECTIONAL_POLICY_TYPE
>>
>> and a fixed port so it can be used as URL.
>>
>> There must be some small window  where the port is already open and ssl handshake can done,  but ssl  malfunctioning and sending wrong data.
>>
>> Special omniorb options used:
>>
>> "serverTransportRule", "* unix,tcp,ssl,bidir", "acceptBiDirectionalGIOP", "1",
>>
>> "clientTransportRule", "* unix,tcp,ssl,bidir", "offerBiDirectionalGIOP", "1",
>>
>> "connectionWatchImmediate", "1",
>>
>> My questions are, is this really an openssl bug? Can something bad happen in omniORB initialization as well?
>>
>> Any ideas on how to debug this further are welcome. I'll try raising traceLevel, but it might be a while until it happens again.
>>
>>
>> Regards,
>>
>>   Michael
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> omniORB-list mailing list
>> omniORB-list at omniorb.net
>> https://www.omniorb-support.com/mailman/listinfo/omniorb-list
>
> _______________________________________________
> omniORB-list mailing list
> omniORB-list at omniorb.net
> https://www.omniorb-support.com/mailman/listinfo/omniorb-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.omniorb-support.com/pipermail/omniorb-list/attachments/20240223/3609c900/attachment-0001.html>