Demand-driven object loading
Sai-Lai Lo
S.Lo@orl.co.uk
Fri, 10 Oct 1997 14:31:28 +0100
>>>>> Randy Shoup writes:
> Sai-Lai --
> We are looking into implementing a demand-driven approach to loading
> persistent objects in omniORB. In looking through the maillist
> archives, I found the following references, where this issue was
> discussed:
> http://www.orl.co.uk/omniORB/archives/1997-Jun/0025.html
> http://www.orl.co.uk/omniORB/archives/1997-Jun/0027.html
> In thinking about the issues, we were concerned that the forwarding
> approach would require us to have a potentially very large number of
> active objects -- at least a reDirector for each available object. We
> are leaning toward the objectLoader approach, most particularly because
> it would allow us to control/minimize the number of active objects in
> the system at any one time. If we were to end up modifying the ORB code
> to do this, what sorts of problems do you think this would cause? How
> problematic are the concurrency issues you allude to in your note?
I agreed the reDirector approach alone would still require one reDirector
for every available object. This is not scalable.
To answer your question, I will first describe the limitations of the
current omniORB implementation. Then I'll outline an interim extension to
the ORB to support an object loader.
Limitations
-----------
1. Object IDs
omniORB_2.2.0 uses fixed size object identifiers (OIDs). It treats its
OIDs as pure names, i.e. it does not divide an OID into subfields and
uses the whole OID for equality test. Applications can specify the OID for
an object implementation but it can only be the same type as those
generated by the ORB. Using fixed size OIDs allow us to do simple and
fast hashing to locate an object.
With large scale persistent object stores, it is quite likely that the
applications would like to name objects using variable/fixed-size OIDs
that cannot fit into the OIDs used by the ORB. There is also the
requirement of loading the objects on demand rather than statically at
startup time. Some form of object loader is desirable.
In general, an application may have multiple backend stores that it can
load objects from and it may want to use different object loaders for
these stores.
2. Object Loaders
To satisfy these requirements, the ORB needs to support another form of
OID:
<object manager ID><internal object ID>
The application registers with the ORB an object manager ID (OMID) and
an object loader to receive upcalls from the ORB. The object loader will
be passed the internal object ID (IOID) and it is its job to further
decode this ID and load the object from the persistent store. Once the
object is instantiated, it pass its reference back to the ORB. The ORB
can then dispatch requests to the object. Notice that this is a slow
path to locate objects that have already been loaded. It is better for
the objects to issue location forward replies to inform the clients of
the fixed size OIDs the ORB uses internally to identify the
objects. Subsequent calls to the objects from these clients will then be
dispatched by the ORB through its normal fast path.
As you know, this is not supported by omniORB_2.2.0. However, this is at
the top of my to-do list and will be done once I get the coming release
out of the door.
Interim Extension
-----------------
(A) Matthew Newhook has added an object loader to omniORB_2.2.0. He has
made his changes available for download:
http://www.orl.co.uk/omniORB/archives/1997-Oct/0013.html
This may be what you want if you are happy to use the fixed size
omniORB OIDs to name your persistent objects.
(B) A more general object loader solution to support application defined
variable length OIDs can be done with small amount of modifications.
I attached at the end the code to illustrate how this can be done.
I'm not completely happy with the solution because it only supports a
single loader. Of course you can have a chain of loaders and try each one
in turn until one succeeds or all fail. That is not satisfactory because
your loaders may be called unnecessarily.
To install an object loader, the application writes a function with this
signature:
typedef CORBA::Object_ptr (*loader)(CORBA::Char*key,CORBA::ULong keysize);
The application instantiates the variable omniORB::objectLoader with the
pointer to this function.
The function should locate the object using the key argument and return
the object implmentation it finds in its own active object table. If the
key does not point to a valid object, it should raise a
CORBA::OBJECT_NOT_EXIST exception. Concurrent upcalls may be dispatched
by the ORB to this function so it has to deal with the concurrency
appropriately.
If the application wants the ORB to dispatch future requests to the
objects through its fast path, the loader should returns a reDirector
object instead of the real implementation object. The reDirectors can
then capture the requests and issue location forward messages to the real
objects. To do this properly, the ORB should assume the role of the
reDirectors and issues location forward messages internally. However,
to do so would require more extensive changes to the ORB.
Have a look. Comments and feedbacks are welcomed.
Regards,
Sai-Lai
------- Interim extension to support application defined object loader ---
1. Add the type definition for the loader and the global variable
omniORB::objectLoader in include/omniORB2/omniORB.h
2. Replace these two functions in src/lib/omniORB2/giopServer.cc with the
following:
XXXXXX One should also do similar modifications to createObjRef() in
src/lib/omniORB2/objectRef.cc. The function is affected if an object
reference unmarshalled from the wire points to a local object that
should be loaded by the loader.
void
GIOP_S::HandleRequest(CORBA::Boolean byteorder)
{
CORBA::ULong msgsize;
CORBA::ULong keysize;
try {
// This try block catches any exception that might raise during
// the processing of the request header
msgsize <<= *this;
if (msgsize > MaxMessageSize()) {
SendMsgErrorMessage();
setStrandDying();
throw CORBA::COMM_FAILURE(0,CORBA::COMPLETED_NO);
}
RdMessageSize(msgsize,byteorder); // Set the size of the message body.
// XXX Do not support any service context yet,
// XXX For the moment, skips the service context
CORBA::ULong svcccount;
CORBA::ULong svcctag;
CORBA::ULong svcctxtsize;
svcccount <<= *this;
while (svcccount-- > 0) {
svcctag <<= *this;
svcctxtsize <<= *this;
skip(svcctxtsize);
};
pd_request_id <<= *this;
pd_response_expected <<= *this;
keysize <<= *this;
get_char_array((CORBA::Char *)&pd_objkey,keysize);
CORBA::ULong octetlen;
octetlen <<= *this;
if (octetlen > OMNIORB_GIOPDRIVER_GIOP_S_INLINE_BUF_SIZE)
{
// Do not blindly allocate a buffer for the operation string
// a poison packet might set this to a huge value
if (octetlen > RdMessageUnRead()) {
throw CORBA::MARSHAL(0,CORBA::COMPLETED_NO);
}
CORBA::Octet *p = new CORBA::Octet[octetlen];
if (!p)
throw CORBA::NO_MEMORY(0,CORBA::COMPLETED_NO);
pd_operation = p;
}
get_char_array((CORBA::Octet *)pd_operation,octetlen);
octetlen <<= *this;
if (octetlen > OMNIORB_GIOPDRIVER_GIOP_S_INLINE_BUF_SIZE)
{
// Do not blindly allocate a buffer for the principal octet vector
// a poison packet might set this to a huge value
if (octetlen > RdMessageUnRead()) {
throw CORBA::MARSHAL(0,CORBA::COMPLETED_NO);
}
CORBA::Octet *p = new CORBA::Octet[octetlen];
if (!p)
throw CORBA::NO_MEMORY(0,CORBA::COMPLETED_NO);
pd_principal = p;
}
get_char_array((CORBA::Octet *)pd_principal,octetlen);
}
catch (CORBA::MARSHAL &ex) {
RequestReceived(1);
SendMsgErrorMessage();
pd_state = GIOP_S::Idle;
return;
}
catch (CORBA::NO_MEMORY &ex) {
RequestReceived(1);
MarshallSystemException(this,SysExceptRepoID::NO_MEMORY,ex);
return;
}
// Note: If this is a one way invocation, we choose to return a
// MessageError Message instead of returning a Reply with System Exception
// message because the other-end says do not send me a reply!
omniObject *obj = 0;
try {
if (omniORB::objectLoader == 0) {
if (keysize != sizeof(omniObjectKey)) {
throw CORBA::OBJECT_NOT_EXIST(0,CORBA::COMPLETED_NO);
}
obj = omni::locateObject(pd_objkey);
}
else {
// The application has installed an object loader
// If we cannot locate the object in our object table, we call
// the object loader to ask for it.
// If it throws OBJECT_NOT_EXIST if it cannot find it either.
// The loader should be capable of dealing with concurrent upcalls.
try {
if (keysize == sizeof(omniObjectKey))
obj = omni::locateObject(pd_objkey);
}
catch (...) { }
if (!obj) {
// Be careful with the reference count!
// The returned object should have the reference count increment
// by 1, like it is done in omni::locateObject()
obj = omniORB::objectLoader(pd_objkey,keysize)->PR_getobj();
}
}
if (!obj->dispatch(*this,(const char *)pd_operation,pd_response_expected))
{
if (!obj->omniObject::dispatch(*this,(const char*)pd_operation,
pd_response_expected))
{
RequestReceived(1);
throw CORBA::BAD_OPERATION(0,CORBA::COMPLETED_NO);
}
}
}
catch (CORBA::OBJECT_NOT_EXIST &) {
if (!obj) {
RequestReceived(1);
if (!pd_response_expected) {
// This is a one way invocation, we choose to return a MessageError
// Message instead of returning a Reply with System Exception
// message because the other-end says do not send me a reply!
SendMsgErrorMessage();
ReplyCompleted();
}
else {
MarshallSystemException(this,SysExceptRepoID::OBJECT_NOT_EXIST,ex);
}
}
else {
CHECK_AND_MAYBE_MARSHALL_SYSTEM_EXCEPTION (OBJECT_NOT_EXIST,ex);
}
}
catch (CORBA::UNKNOWN &ex) {
CHECK_AND_MAYBE_MARSHALL_SYSTEM_EXCEPTION (UNKNOWN,ex);
}
catch (CORBA::BAD_PARAM &ex) {
CHECK_AND_MAYBE_MARSHALL_SYSTEM_EXCEPTION (BAD_PARAM,ex);
}
catch (CORBA::NO_MEMORY &ex) {
CHECK_AND_MAYBE_MARSHALL_SYSTEM_EXCEPTION (NO_MEMORY,ex);
}
catch (CORBA::IMP_LIMIT &ex) {
CHECK_AND_MAYBE_MARSHALL_SYSTEM_EXCEPTION (IMP_LIMIT,ex);
}
catch (CORBA::COMM_FAILURE &ex) {
setStrandDying();
CHECK_AND_MAYBE_MARSHALL_SYSTEM_EXCEPTION (COMM_FAILURE,ex);
}
catch (CORBA::INV_OBJREF &ex) {
CHECK_AND_MAYBE_MARSHALL_SYSTEM_EXCEPTION (INV_OBJREF,ex);
}
catch (CORBA::NO_PERMISSION &ex) {
CHECK_AND_MAYBE_MARSHALL_SYSTEM_EXCEPTION (NO_PERMISSION,ex);
}
catch (CORBA::INTERNAL &ex) {
CHECK_AND_MAYBE_MARSHALL_SYSTEM_EXCEPTION (INTERNAL,ex);
}
catch (CORBA::MARSHAL &ex) {
CHECK_AND_MAYBE_MARSHALL_SYSTEM_EXCEPTION (MARSHAL,ex);
}
catch (CORBA::INITIALIZE &ex) {
CHECK_AND_MAYBE_MARSHALL_SYSTEM_EXCEPTION (INITIALIZE,ex);
}
catch (CORBA::NO_IMPLEMENT &ex) {
CHECK_AND_MAYBE_MARSHALL_SYSTEM_EXCEPTION (NO_IMPLEMENT,ex);
}
catch (CORBA::BAD_TYPECODE &ex) {
CHECK_AND_MAYBE_MARSHALL_SYSTEM_EXCEPTION (BAD_TYPECODE,ex);
}
catch (CORBA::BAD_OPERATION &ex) {
CHECK_AND_MAYBE_MARSHALL_SYSTEM_EXCEPTION (BAD_OPERATION,ex);
}
catch (CORBA::NO_RESOURCES &ex) {
CHECK_AND_MAYBE_MARSHALL_SYSTEM_EXCEPTION (NO_RESOURCES,ex);
}
catch (CORBA::NO_RESPONSE &ex) {
CHECK_AND_MAYBE_MARSHALL_SYSTEM_EXCEPTION (NO_RESPONSE,ex);
}
catch (CORBA::PERSIST_STORE &ex) {
CHECK_AND_MAYBE_MARSHALL_SYSTEM_EXCEPTION (PERSIST_STORE,ex);
}
catch (CORBA::BAD_INV_ORDER &ex) {
CHECK_AND_MAYBE_MARSHALL_SYSTEM_EXCEPTION (BAD_INV_ORDER,ex);
}
catch (CORBA::TRANSIENT &ex) {
CHECK_AND_MAYBE_MARSHALL_SYSTEM_EXCEPTION (TRANSIENT,ex);
}
catch (CORBA::FREE_MEM &ex) {
CHECK_AND_MAYBE_MARSHALL_SYSTEM_EXCEPTION (FREE_MEM,ex);
}
catch (CORBA::INV_IDENT &ex) {
CHECK_AND_MAYBE_MARSHALL_SYSTEM_EXCEPTION (INV_IDENT,ex);
}
catch (CORBA::INV_FLAG &ex) {
CHECK_AND_MAYBE_MARSHALL_SYSTEM_EXCEPTION (INV_FLAG,ex);
}
catch (CORBA::INTF_REPOS &ex) {
CHECK_AND_MAYBE_MARSHALL_SYSTEM_EXCEPTION (INTF_REPOS,ex);
}
catch (CORBA::BAD_CONTEXT &ex) {
CHECK_AND_MAYBE_MARSHALL_SYSTEM_EXCEPTION (BAD_CONTEXT,ex);
}
catch (CORBA::OBJ_ADAPTER &ex) {
CHECK_AND_MAYBE_MARSHALL_SYSTEM_EXCEPTION (OBJ_ADAPTER,ex);
}
catch (CORBA::DATA_CONVERSION &ex) {
CHECK_AND_MAYBE_MARSHALL_SYSTEM_EXCEPTION (DATA_CONVERSION,ex);
}
catch (...) {
CORBA::UNKNOWN ex(0,CORBA::COMPLETED_MAYBE);
CHECK_AND_MAYBE_MARSHALL_SYSTEM_EXCEPTION (UNKNOWN,ex);
}
if (obj)
omni::objectRelease(obj);
return;
}
#undef CHECK_AND_MAYBE_MARSHALL_SYSTEM_EXCEPTION
void
GIOP_S::HandleLocateRequest(CORBA::Boolean byteorder)
{
CORBA::ULong msgsize;
CORBA::ULong keysize;
try {
// This try block catches any exception that might raise during
// the processing of the request header
msgsize <<= *this;
if (msgsize > MaxMessageSize()) {
SendMsgErrorMessage();
setStrandDying();
throw CORBA::COMM_FAILURE(0,CORBA::COMPLETED_NO);
}
RdMessageSize(msgsize,byteorder); // Set the size of the message body.
pd_request_id <<= *this;
keysize <<= *this;
get_char_array((CORBA::Char *)&pd_objkey,keysize);
RequestReceived();
}
catch (CORBA::MARSHAL &ex) {
RequestReceived(1);
SendMsgErrorMessage();
pd_state = GIOP_S::Idle;
return;
}
catch (CORBA::NO_MEMORY &ex) {
RequestReceived(1);
MarshallSystemException(this,SysExceptRepoID::NO_MEMORY,ex);
return;
}
omniObject *obj = 0;
GIOP::LocateStatusType status;
try {
if (omniORB::objectLoader == 0) {
if (keysize != sizeof(omniObjectKey)) {
throw CORBA::OBJECT_NOT_EXIST(0,CORBA::COMPLETED_NO);
}
obj = omni::locateObject(pd_objkey);
}
else {
// The application has installed an object loader
// If we cannot locate the object in our object table, we call
// the object loader to ask for it.
// If it throws OBJECT_NOT_EXIST if it cannot find it either.
// The loader should be capable of dealing with concurrent upcalls.
try {
if (keysize == sizeof(omniObjectKey))
obj = omni::locateObject(pd_objkey);
}
catch (...) { }
if (!obj) {
// Be careful with the reference count!
// The returned object should have the reference count increment
// by 1, like it is done in omni::locateObject()
obj = omniORB::objectLoader(pd_objkey,keysize)->PR_getobj();
}
}
omni::objectRelease(obj);
status = GIOP::OBJECT_HERE;
// XXX what if the object is relocated. We should do GIOP::OBJECT_FORWARD
// as well.
}
catch(...) {
status = GIOP::UNKNOWN_OBJECT;
}
WrLock();
pd_state = GIOP_S::ReplyIsBeingComposed;
size_t bodysize = 8;
WrMessageSize(0);
put_char_array((CORBA::Char *)MessageHeader::LocateReply,
sizeof(MessageHeader::LocateReply));
operator>>= ((CORBA::ULong)bodysize,*this);
operator>>= (pd_request_id,*this);
operator>>= ((CORBA::ULong)status,*this);
flush();
pd_state = GIOP_S::Idle;
WrUnlock();
return;
}
3. This is the hairy bit. You have to change the marshal object reference
code so that it inserts the (variable length) persistent OID into the
IOR instead of the fixed length internal OID generated by the ORB.
The ORB code assumes that if an object is not a proxy, its OID is
always the fixed length OID, so be careful. If you are interested, I'll
give it some thought as to how to do this properly.