[omniORB] ILU to omniORB (lengthy)

[Geoff Gerrietts]

Quoting Duncan Grisby (duncan at grisby.org):
> I assume you mean using the omniORB.any.to_any and from_any
> functions. That isn't what they are intended for, so I wouldn't
> recommend using them. Just write a wrapper class around the object
> references you use that translates arguments to and from a sequence
> of structs of key value pairs.

I think that's what we settled on.

> It is a very bad idea to touch or directly use the _objref_ classes.
> They are internal to omniORB and might change.

Got it.

> > 3.  omniidl's python backend makes some assumptions....
> >     The omniidl stubber drops a file Module_idl.py and creates Module
> >     and Module_POA packages, all in the current directory.
> 
> To be precise, for IDL file foo.idl, containing module Bar, it outputs
> foo_idl.py and packages Bar and Bar__POA. All of these can be put in
> different packages using combinations of -Wbpackage, -Wbmodules and
> -Wbstubs.

I was able to make these build properly, using the following incantation.

  pushd ~/src ; \
  omniidl -bpython -Iservices/common -Wbpackage=services.$(FOLDER) \
  services/$(FOLDER)/$@.idl ; \
  popd

It just took some experimenting with the backend's package option,
which I previously hadn't connected with my problem because I want the
stuff generated in the current working directory.

> Out of interest, what does ILU do?  How do you find the modules it
> outputs if they aren't on PYTHONPATH?

All ILU's files are generated into the current directory. These import
each other just fine using the "implied relative path" feature of
import. We could then generate these files into whatever deeply-nested
package hierarchy we wanted -- we could in fact pick the local
directory up after generation and move it anywhere in the package
hierarchy without screwing up generated references. That's not a huge
feature, it's just a different way of doing things.

> If you have lots of different bits of IDL to deal with, and you
> don't want to have to decide where to put them, maybe you'd be
> better off compiling it on the fly using omniORB.importIDL().

An interesting thought, and one that I hadn't even considered or
recognized as an option before. I think it's probably not necessary
for us, though -- and now that I've solved the problem in the
makefile, I'm less inclined to try to solve it better. :)

> What kinds of things do you want to monitor?  For most purposes,
> logging from omniORB plus logging from your application (use the
> logging module) are sufficient.

We pump a lot of volume through our services, and some of the services
are very heavily laden with tasks. I can't provide concrete numbers,
but I'm estimating somewhere in the thousands of transactions per
minute. Under ILU, we would occassionally see failures of one sort or
another. Again, I don't have specifics, but under that kind of load,
you sometimes expose unusual bugs, all of which are beyond the
expertise and manpower of our enterprise to resolve. So we monitor for
those events and restart the servers when they stop working.

One of the big stumbling blocks in this conversion project has been
figuring out how to get on top of that again. Yes, it's nice that
omniORB has a lively community and we can probably get someone to at
least explain how to go about tracking down the problem so it can be
really fixed instead of just worked around. That's a strong point of
appeal. But we're also interested in improving the quality of our
monitoring.

I read in the archives of the list about someone who was doing some
monitoring where they calculate a decaying average of processing time.
I'm not sure how these folks were calculating processing time; I know
they were reporting it by having all their interfaces inherit a common
monitoring interface. That's roughly the approach I was intending to
go after: each server has an interface that implements Monitor, and
Monitor contains a heartbeat() method.

But that's just what I'm planning. I don't delude myself into
believing it's the best possible. I'm looking for ideas and insights
into keeping omniORB servers alive and functional in a heavy-load
environment -- and in particular, in knowing when they are not, or
under heavy load, or whatever else I might be able to keep an eye on.

Watching ps is okay, and watching the logs is okay too. But we don't
have someone who can do that and only that all day long, so we hafta
write some tools to do it for us, as best we can.

Thanks,
--G.

-- 
Geoff Gerrietts             "Don't get suckered in by the comments-- 
<geoff at gerrietts net>     they can be terribly misleading.
www.gerrietts.net/geoff/     Debug only code." --Dave Storer