Subtleties of Atomic Loading

Most of the time, atomic loading works like a charm.  Every once in a while, though, it gives you very confusing results.

I ran into one of these with the GemKit upgrade to VisualWorks 7.8.  I'd already determined that the package defined a compiler along with a class that used that compiler in one package.  That's a no-no.  I was getting into infinite loops that way.  I split the methods compiled by the new compiler into another package and that seemed to fix the problem.

The key phrase there is "seemed to".  I started finding that I was getting Message Not Understood errors by things that were supposed to be references to classes but were references to namespaces instead.  For example, code that read Core.Array was being compiled as if it was just Core.  I looked at how it was being compiled and it turned out that it was using the new compiler defined in the previous package.  The problem was that it wasn't supposed to use that compiler.  It was supposed to use the standard Smalltalk compiler.

As it turns out, there's code in the compilerClass method of the class that enables the special compiler on subclasses of this class, not on this class itself.  The code looked like this:

GBC.GbcHierarchyRoot class method:

compilerClass
 ^self  == GBC.GbcHierarchyRoot
  ifTrue: [super compilerClass]
  ifFalse: [GBC.GbcCompiler]

So, if the method being compiled is in GbcHierarchyRoot itself use the default compiler, otherwise, use the GbcCompiler for the subclass methods.

The problem here is that when you're doing atomic loading, at the time that this method is run, the class is in a different namespace than GBC.GbcHierarchyRoot. As such, the test was failing and using the GbcCompiler even for the methods in GbcHierarchyRoot itself.

The solution to this problem was to change the method as follows:

compilerClass

 ^name = #GbcHierarchyRoot
  ifTrue: [super compilerClass]
  ifFalse: [GBC.GbcCompiler]

Now the test works properly and it uses the default compiler class for GbcHierarchyRoot methods.  There were several methods that used this pattern, so I changed them all.

I'm making some progress but it's slow and complicated working out the interactions between atomic loading and the GemKit code. My goal is to keep the atomic analysis loader turned on all the time and still be able to load GemStone code from Store.  So far, I seem to be able to at least load the GemKit code.  The next step is to load actual GemStone code from Store.

GemKit and Atomic Loading revisited

Ok, I spoke too soon.  It turns out that the loading of GbcManagement didn't actually work properly.  There were 26 methods in GbcHierarchyRoot which didn't install.  After a bit of tracing, I discovered why.  The GbcHierarchyRoot class defines a new compiler for instance and class methods and that compiler was loaded in the same package as the GbcHierarchyRoot class.  This means that GbcHierarchyRoot could load before the compiler, load the methods to change to the compiler that's not loaded, then fail to compile anything else.

The solution to this is to create another package which has all the GbcHierarchyRoot methods and is loaded after GbcManagement.  This allows everything to load cleanly.

Now, on to more problems...

GemKit and Atomic Loading

In VisualWorks 7.8, I have to turn off atomic loading in order to load the package GbcManagement. If I didn't, the load crashed with a memory overflow.

After tracing down the problem, I found that the load itself actually worked fine.  It was the step right afterward that did a purgeUnusedBindings that was failing - specifically in a method called relinkSystem.  The relinkSystem method looped through allInstances of DeferredBinding.  Maybe it was hitting an object that was garbage but hadn't yet been garbage collected.  So, I added a call to do a garbage collect just before the allInstances.  That did the trick.  GbcManagement loads perfectly every time now.

It goes to show that you have to be careful with allInstances.

GemKit in VW7.8

So, I have GemKit running in 7.5.  I think I need to get newer versions of GemBuilder/S but it basically runs.  I thought I'd try loading it into 7.8 to see what happens.

It took several tries to get GemKit loaded into 7.8.  I learned that:

• You can't load Searchlight before loading GemKit.  There's an interaction that causes problems.  I'll investigate that later.
• You must turn off Atomic Loading to load GemKit. When Atomic Loading was turned on, I got into an ugly infinite loop that I couldn't interrupt and that allocated memory until it ran out and crashed.  It wasn't a pretty picture

In the end, I was able to get it loaded and get some of the basic things working.  I've also gone through and looked at all the base overrides and base extensions.  I have agreement in principle from Cincom that they would work with  me to reduce the number of overrides by adding code and hooks to the base where it makes sense and where it doesn't impact the design of the base system.

I can see already that my challenges in this project will be:

• Getting GemKit working with Atomic Loading
• Resolving the conflict between GemKit and Searchlight
• Adapting GemKit to the new Glorp interface to Store
• Adapting GemKit to the new Store Repository browsing tools.  The classes that GemKit used to extend to do this are no longer there.
• Adapting to the newest changes to the merge tool
• Getting Refactorings to work on GemStone code (if this is even possible)
• Performance

So far, though, I feel like I'm off to a good start.

Loading GemKit

So, I've been able to get a 32 bit GemStone database running and connect to it from GemBuilder.  The next step is to load GemKit in VW7.5.  This is a fairly straightforward process.  The packages are all in the Cincom Public Store Repository.  You need to load them in the following order.

• GbcManagement 4.0 0.48.0
• GbcReleaseTools 4.0 062.0
• GS_AllUsers_SystemUser  empty 04.0
• GS_Globals empty 11.0
• GS_GemKit 4.0 000.0

Next, I need to push the methods in GS_GemKit to GemStone.  There aren't many methods, so I just connected into GemStone, opened a GemStone browser, and just used copy/paste from the VW browser to the GemStone browser to move all the methods over.  If anyone has a better way of doing this, please let me know.  I then commit my changes.

With all of these packages loaded in, I need to populate my GS_Globals package with the classes in GemStone.  I first connect into GemStone with my user id and open a VisualWorks browser.  From the browser, I select GS_Globals and from a context menu select "GemKit Compare With GemStone"  This gave me a window that allowed me to select all the methods and click on "Update Image".  It seems a bit odd that it asks me to define shared variables called Nil, True and False.  I say no to each one and the load finishes without any problems.  That question about shared variables is mentioned in the documentation method GbcSourceManager class >> documentation_installation.  It's also described in the method GbcSharedVariable >> methodsNeverToDefineInClient where the array #(#nil #true #false) is commented out and the returned answer is just #().  It might be best if we can keep the original code here.  I'll investigate.

That's pretty much it.  I've been able to push code back and forth to GemStone and it seems to be working reasonably well.

Next up:  My first attempts with GemKit in VW7.8

Getting started with GemKit - part 1

I've started working on upgrading GemKit to VisualWorks 7.8.  The first order of business is to get GemKit running in the latest environment known to work.  That would be VisualWorks 7.5.  Although that version is reported to be working in VisualWorks 7.6, it hasn't been tested much in that environment so I figured that 7.5 is a better starting point.

Before I can get GemKit running, though, I need GemStone.  I have two choices - I can run the 32 bit version on my Windows laptop which is the most convenient or I can run the 64 bit version on my 64-bit Ubuntu Linux system.  Unfortunately, the processor on my laptop doesn't have the ability to run VMWare so I can't run a virtual Linux OS on the laptop.

I decide that I actually need both 32 and 64 bit GemStone versions and I'd like to make sure GemKit runs on both. So, while I upgrade Linux to the latest version of Ubuntu, I grab GemStone 6.5.8 and install it.

Installing GemStone on Windows takes some effort but if you follow the instructions carefully you can do it. One step of the instructions, however, is that you need to put your gemstone.key file into the sys directory.  Lovely.  Where do I get gemstone.key?  Good news! There's a key in the Cincom installation directory for VW7.8 under contributed/GemStone/Server/Windows.  Yay!  Bad news: the key expires April 15th, 2010 - about a year before VW7.8 was released.  Fortunately, I can fool GemStone by setting my system clock back.  It's not pretty but it works.

Ok, I have GemStone 6.5.8 running.  Now I need GemKit.  Fortunately, there are parcels available in the contributed directory.  Ok, that's another problem.  Who would have thought that the contributed directory of VisualWorks would have 32 bit version of the server but 64 but versions of GemBuilder Smalltalk?  I'm sure that it will come in handy when I connect to the Linux system, but for now, I need to find a 32 bit version of GemBuilder.

The GemStone community site allows you to download lots of versions of GemBuilder/Smalltalk.  All of them are 64 bit versions, though.  I finally managed to track down 32 bit versions in the Cincom Public Store Repository.  These seem like older versions but with a few tweaks and fixes I can get them to connect to GemStone 6.5.8.  Yay!

Installing GemStone 64 on Ubuntu Linux is a bit more challenging.  One part of the process involves verifying the checksums of all the files.  This process fails left right and center with error messages.  I suspect that this is because the installation instructions are for RedHat Linux and not Ubuntu.  Fortunately, I can skip this step and continue to install the files.  When I get to the point where I need a licence key, I decide to put it aside for now.  I'm working through my channels to get valid license keys for everything.

In part 2, I'll talk about getting GemKit running.