Atomic Loading working with GemKit

After much pain mucking about in namespaces, I finally managed to load GemStone code saved in Store using atomic loading.

Atomic loading first loads code into a temporary namespace called the shadow namespace.  Once everything is compiled there and it looks good, the system is re-compiled into the real namespace and the shadow namespace disappears.

With GemStone code, I thought I had everything setup to compile the code into the shadow namespace but when I did the actual compile, the code ended up in the real namespace right away.  This is bad because in the shadow world, the system would add the compilerClass and classCompilerClass methods to tell the shadow class what the compiler is.  You don't want to move these methods into the real namespace because it would cause crashes - which it did.

To allow me to quickly see what was going on, I wrote a tool that simply showed a tree view with a portion of the namespace hierarchy that I was interested in along with the shadow hierarchy.  It also showed the identityHashes of the namespaces so I could quickly tell what namespace I was looking at in an inspector.  If the namespace contained shared variables or classes, it would show those in the tree and if the classes had methods, it would show those as well.  Here's what it looked like.

Now for the problem.  When you login to GemStone as a user (for example, David), GemKit creates a namespace for you called Root.GemStone.AllUsers.David.  It also aliases this as Root.GemStone.CurrentUser. You can see that in the above image.

When you create the shadow environment, Store creates a mimic of the namespace structure.  Since CurrentUser wasn't marked as a namespace link, Store didn't create it in Shadow. Ok, fair enough.  I augmented the system to create CurrentUser in the shadow namespace. Sadly, that wasn't enough to fix the problem.

Now when I try to compile, the system locates Root.GemStone in the shadow namespace (so far so good) and asks it to resolve the symbol #UserGlobals.  This is where it will put the class it's compiling.  UserGlobals isn't in GemStone directly but rather in CurrentUser which is in the imports of GemStone. The problem:  The imported namespace CurrentUser is coming from the real environment instead of the shadow environment.  It finds UserGlobals in the real world and compiles the class directly there.

Aaarg - ok, I go and smash the imports in the shadow GemStone namespace to point to the CurrentUser in the shadow.  Now it complains about a method conflict during the install.  When you try to move the shadow classes to the real environment, you must first move the class then move the methods.  When you move a GemStone class to the real world, you automatically get a class method called gsst_definition.  When we try to move the gsst_definition method from the shadow world to the real world, the system detects that there's a conflict and stops.

As it turns out, you have to change both imported namespaces in the shadow GemStone to point to the shadow and you also need to change the imported namespaces in the shadow GemStone.Globals to point back to the shadow GemStone instead of the real GemStone.

<sigh>  After all that, atomic loading works with GemStone code.  Yay!  I think I'll pop a bottle of bubbly and share it with my wife tonight.

For my next challenge:  Cincom switched Store to use Glorp instead of the older direct mechanism.  GemKit has tools which are coded for the old mechanism and need to be modified to work with Glorp.

GemKit for VisualWorks 7.8 Preview

I have a GemKit for VisualWorks 7.8 preview available at the following URL:

    http://simberon.com/downloads/GemKit-VW78-Preview.zip

Follow the instructions in the readme.txt to load it.  Please report any issues to david@simberon.com.

GemKit and Atomic Loading revisited

Ok, I spoke too soon.  It turns out that the loading of GbcManagement didn't actually work properly.  There were 26 methods in GbcHierarchyRoot which didn't install.  After a bit of tracing, I discovered why.  The GbcHierarchyRoot class defines a new compiler for instance and class methods and that compiler was loaded in the same package as the GbcHierarchyRoot class.  This means that GbcHierarchyRoot could load before the compiler, load the methods to change to the compiler that's not loaded, then fail to compile anything else.

The solution to this is to create another package which has all the GbcHierarchyRoot methods and is loaded after GbcManagement.  This allows everything to load cleanly.

Now, on to more problems...

GemKit and Atomic Loading

In VisualWorks 7.8, I have to turn off atomic loading in order to load the package GbcManagement. If I didn't, the load crashed with a memory overflow.

After tracing down the problem, I found that the load itself actually worked fine.  It was the step right afterward that did a purgeUnusedBindings that was failing - specifically in a method called relinkSystem.  The relinkSystem method looped through allInstances of DeferredBinding.  Maybe it was hitting an object that was garbage but hadn't yet been garbage collected.  So, I added a call to do a garbage collect just before the allInstances.  That did the trick.  GbcManagement loads perfectly every time now.

It goes to show that you have to be careful with allInstances.

GemKit in VW7.8

So, I have GemKit running in 7.5.  I think I need to get newer versions of GemBuilder/S but it basically runs.  I thought I'd try loading it into 7.8 to see what happens.

It took several tries to get GemKit loaded into 7.8.  I learned that:

• You can't load Searchlight before loading GemKit.  There's an interaction that causes problems.  I'll investigate that later.
• You must turn off Atomic Loading to load GemKit. When Atomic Loading was turned on, I got into an ugly infinite loop that I couldn't interrupt and that allocated memory until it ran out and crashed.  It wasn't a pretty picture

In the end, I was able to get it loaded and get some of the basic things working.  I've also gone through and looked at all the base overrides and base extensions.  I have agreement in principle from Cincom that they would work with  me to reduce the number of overrides by adding code and hooks to the base where it makes sense and where it doesn't impact the design of the base system.

I can see already that my challenges in this project will be:

• Getting GemKit working with Atomic Loading
• Resolving the conflict between GemKit and Searchlight
• Adapting GemKit to the new Glorp interface to Store
• Adapting GemKit to the new Store Repository browsing tools.  The classes that GemKit used to extend to do this are no longer there.
• Adapting to the newest changes to the merge tool
• Getting Refactorings to work on GemStone code (if this is even possible)
• Performance

So far, though, I feel like I'm off to a good start.

Loading GemKit

So, I've been able to get a 32 bit GemStone database running and connect to it from GemBuilder.  The next step is to load GemKit in VW7.5.  This is a fairly straightforward process.  The packages are all in the Cincom Public Store Repository.  You need to load them in the following order.

• GbcManagement 4.0 0.48.0
• GbcReleaseTools 4.0 062.0
• GS_AllUsers_SystemUser  empty 04.0
• GS_Globals empty 11.0
• GS_GemKit 4.0 000.0

Next, I need to push the methods in GS_GemKit to GemStone.  There aren't many methods, so I just connected into GemStone, opened a GemStone browser, and just used copy/paste from the VW browser to the GemStone browser to move all the methods over.  If anyone has a better way of doing this, please let me know.  I then commit my changes.

With all of these packages loaded in, I need to populate my GS_Globals package with the classes in GemStone.  I first connect into GemStone with my user id and open a VisualWorks browser.  From the browser, I select GS_Globals and from a context menu select "GemKit Compare With GemStone"  This gave me a window that allowed me to select all the methods and click on "Update Image".  It seems a bit odd that it asks me to define shared variables called Nil, True and False.  I say no to each one and the load finishes without any problems.  That question about shared variables is mentioned in the documentation method GbcSourceManager class >> documentation_installation.  It's also described in the method GbcSharedVariable >> methodsNeverToDefineInClient where the array #(#nil #true #false) is commented out and the returned answer is just #().  It might be best if we can keep the original code here.  I'll investigate.

That's pretty much it.  I've been able to push code back and forth to GemStone and it seems to be working reasonably well.

Next up:  My first attempts with GemKit in VW7.8

Getting started with GemKit - part 1

I've started working on upgrading GemKit to VisualWorks 7.8.  The first order of business is to get GemKit running in the latest environment known to work.  That would be VisualWorks 7.5.  Although that version is reported to be working in VisualWorks 7.6, it hasn't been tested much in that environment so I figured that 7.5 is a better starting point.

Before I can get GemKit running, though, I need GemStone.  I have two choices - I can run the 32 bit version on my Windows laptop which is the most convenient or I can run the 64 bit version on my 64-bit Ubuntu Linux system.  Unfortunately, the processor on my laptop doesn't have the ability to run VMWare so I can't run a virtual Linux OS on the laptop.

I decide that I actually need both 32 and 64 bit GemStone versions and I'd like to make sure GemKit runs on both. So, while I upgrade Linux to the latest version of Ubuntu, I grab GemStone 6.5.8 and install it.

Installing GemStone on Windows takes some effort but if you follow the instructions carefully you can do it. One step of the instructions, however, is that you need to put your gemstone.key file into the sys directory.  Lovely.  Where do I get gemstone.key?  Good news! There's a key in the Cincom installation directory for VW7.8 under contributed/GemStone/Server/Windows.  Yay!  Bad news: the key expires April 15th, 2010 - about a year before VW7.8 was released.  Fortunately, I can fool GemStone by setting my system clock back.  It's not pretty but it works.

Ok, I have GemStone 6.5.8 running.  Now I need GemKit.  Fortunately, there are parcels available in the contributed directory.  Ok, that's another problem.  Who would have thought that the contributed directory of VisualWorks would have 32 bit version of the server but 64 but versions of GemBuilder Smalltalk?  I'm sure that it will come in handy when I connect to the Linux system, but for now, I need to find a 32 bit version of GemBuilder.

The GemStone community site allows you to download lots of versions of GemBuilder/Smalltalk.  All of them are 64 bit versions, though.  I finally managed to track down 32 bit versions in the Cincom Public Store Repository.  These seem like older versions but with a few tweaks and fixes I can get them to connect to GemStone 6.5.8.  Yay!

Installing GemStone 64 on Ubuntu Linux is a bit more challenging.  One part of the process involves verifying the checksums of all the files.  This process fails left right and center with error messages.  I suspect that this is because the installation instructions are for RedHat Linux and not Ubuntu.  Fortunately, I can skip this step and continue to install the files.  When I get to the point where I need a licence key, I decide to put it aside for now.  I'm working through my channels to get valid license keys for everything.

In part 2, I'll talk about getting GemKit running.