Friday, January 14, 2011

Two-sided commit

I've encountered an interesting problem for which I've come up with an interesting solution.

My application needs to read records from several input queues and write thode records into a database. The input queues aren't in the database and have their own commit and abort abilities.  The problem is that to avoid losing records, I have to coordinate the commits.  I need to:
  1. Read a record from one of the queues
  2. Write the object into the database
  3. Commit the database write
  4. Commit the read from the queue
If step 3 fails, then we need to abort in step 4 instead of committing. If we do the commits out of order, we could remove a record from the queue but not be able to do anything with it

I'd like to isolate knowledge of multiple queues from the caller who writes to the database:

   [message := queueManager getMessage.
   message notNil] whileTrue: [
      self beginTransaction.
      self writeMessageIntoDatabase: message.
      self commitTransaction
         ifTrue: [????  commit]
         ifFalse: [???? abort]]

The problem is that I don't know which queue I took the message from - the queue manager isolates me from that knowledge.  But I need that knowledge so I can commit or abort the proper queue later.

What are my options?  I could have getMessage return both a message and a queue. Then I'd know which queue to commit.  It kind of breaks my encapsulation, though.  I still need my method to know about the existence of multiple queues and how to commit or abort them.

I could have the queueManager keep track of which queue it pulled the message from so I could tell the queueManager to commit or abort and it would then tell the appropriate queue to commit or abort.  This seems really messy, though.

My solution is to do this:

   queueManager withAllMessagesDo: [:message |
      self beginTransaction.
      self writeMessageIntoDatabase: message.
      self commitTransaction]

The queueManager delegates the message as-is to each of the queues.  Each queue can then extract a message and run the block on it.  If it returns true, the queue commits and if it returns false, the queue aborts.

The nice thing about this solution is that all the knowledge of multiple queues and even knowledge of how to commit or abort the queues is encapsulated into the queues and doesn't affect the high level code that needs to write to the database.

5 comments:

  1. You don't have a 2-phase commit available?

    ReplyDelete
  2. No, unfortunately a two-phase commit isn't an option. One side is IBM MQ and the other side is GemStone.

    ReplyDelete
  3. Neat. This is actually the 'multiple return values refactoring' plus some extra refactorings.

    step 0: the problem
    [message := queueManager getMessage.
    message notNil] whileTrue: [
    self beginTransaction.
    self writeMessageIntoDatabase: message.
    self commitTransaction
    ifTrue: [???? commit]
    ifFalse: [???? abort]]

    step 1: the multiple return values refactoring
    queueManager hasMessages whileTrue: [
    queueManager getMessage: [:message
    self beginTransaction.
    self writeMessageIntoDatabase: message.
    commitFlag := self commitTransaction]
    andCommit: [commitFlag]]

    of course that's ugly and I'm not even sure it'd work at all. What's important though is that you took the multiple return values (the message and the handle to do the commit) and instead pushed the code that consumes those return values forward onto the class that produced those multiple returns in the first place. So you turned multiple returns back into multiple blocks FORWARD.

    step 2: making it work
    queueManager hasMessages whileTrue: [
    queueManager getMessageAndCommit: [:message
    self beginTransaction.
    self writeMessageIntoDatabase: message.
    self commitTransaction]

    step 3: making it right
    queueManager withAllMessagesDoAndCommit: [:message
    self beginTransaction.
    self writeMessageIntoDatabase: message.
    self commitTransaction]

    which is just a tiny bit better named than yours.

    ReplyDelete
  4. I first learned that refactoring in Building 3D Video Games Using Smalltalk, http://anthony.etherealplanet.org/resources/games-talk.pdf at the very end in Cool Smalltalk Tricks.

    ReplyDelete
  5. It's interesting that you mentioned Anthony's "multiple return values" trick because that was the thought process I went through in coming up with this technique. I thought I needed to return two things - the message and the queue; but wait, I can pass them forward using a two-parameter block; but wait, I don't need to call the queue here because I can push the commit backward to the queue and hide it from the high level code.

    ReplyDelete