[omniORB] Crash in VS8 with static runtime

[Arne Pajunen]

Hi,

I just thought I'd mention that it seems to be very easy to work around 
this issue. The affected pieces are just a homemade dynamic_cast, so if 
you have RTTI support (like vs2005), you can just implement the downcast 
operations with dynamic_cast. The pieces to modify are the downcast 
member in omniIdentity, omniLocalIdentity, omniShutdownIdentity, 
omniInProcessIdentity, omniRemoteIdentity and omniObjTableEntry. I also 
changed the thisClassCompares to use dynamic_cast for good measure, 
although this doesn't seem necessary. To keep the fallback 
functionality, just put the dynamic_cast implementation in #ifdef 
_CPPRTTI blocks (maybe && _MSC_VER >= 1400) and it will fallback to old 
behavior with no RTTI.

Any thoughts on this ? I'm currently testing this and it seems to work 
without problems, if anyone needs it I can put these (minor) changes in 
a diff and post here next week.

ps. There was also some fishy looking code in omniObjTableEntry's 
thisClassCompare (omniInternal.cc), it cast it to either 
omniObjTableEntry, or fallback to omniLocalIdentity and the cast that to 
omniObjTableEntry with c-style cast. I don't know what the intended 
behavior of that is so I just duplicated the effect with 
dynamic_casting, maybe someone else knows what the part is supposed to do?

Best Regards,
Arne Pajunen

Arne Pajunen wrote:
> Hi,
> 
> I found that OmniOrb 4.0.7 stable crashes when compiled with Visual
> studio 2005 (vs8) static runtime (/MT) under certain circumstances.
> 
> The problem manifested for me when resolving NameService (testcase
> source code attached) with a crash in release mode that disappears when
> compiling for debug, or even with debug information. The problem
> occurred when both OmniOrb libraries and the program were compiled with
> /MT runtime, and disappeared sporadically.
> 
> I've tracked the cause to be the linker option "/OPT:REF" which gets
> enabled by default for release builds, and disabled in debug. It enabled
> "identical COMDAT folding", which to my understanding removes identical
> functions and keeps just one. This causes a crash because it folds some
> things it shouldn't, as you can see from the attached verbose linker
> output snippet. It replaces the omniIdentity classes (Local,Remote,..)
> thisClassCompare with a single copy, and because the classes are
> identified by the function pointer (if I understood correctly), all
> comparisons become true.
> 
> This causes the crash in my scenario in createObjRef (omniInternal.cc),
> as you can see from the attached stacktrace (minimal) and program
> outputs. It falsely identifies remoteIdentities as local and promptly
> crashes.
> 
> As a workaround, you can build projects with static runtime that link
> OmniOrb libraries with /OPT:NOICF , which disables the duplicate removal
> and makes the code work. However, I think it would be beneficial in the
> long run to change this code so it compiles correctly with /OPT:ICF
> which is a default for new projects in VS2005.
> 
> Although I detected this in 4.0.7 and haven't tried newer versions, I
> believe omniORB-4.1.0-beta2 to be affected as well because the relevant
> sections of code seem similar.
> 
> I would be most interested in any thoughts, and ideas for a patch.
> 
> Best regards,
> 
-- 
Arne Pajunen
Software Engineer
OpenTTCN Oy, Test and Test Control Components for Test System Vendors
   Web: http://www.openttcn.com
Email: arne.pajunen at openttcn.fi