Ok, let's see a show of hands - how many people have GemStone projects running on 6.1.5? Ok, of those, how many encountered a mysterious crash yesterday?
It seems that RcQueue uses a timestamp for its records. That timestamp is the number of seconds since January 1, 1995. It's constrained to be a small integer. Yesterday, it overflowed to a large integer, violated the constraint, and brought our production server to its knees.
Fortunately, there's a fix in GemStone 6.6 - it now records timestamps since January 1, 2005. There, that fixes the problems. Everybody set your watches for January 5, 2022.
Dennis Smith had the same problem in one of our very old applications.
ReplyDeleteIt pays to upgrade! 6.1.5 was released in January 2006 (just a tad out of date).
ReplyDeleteWe are actually in the last phases of upgrading to GemStone 6.6. We're in the testing phase and should have it upgraded in a month or two. It's too bad it happened just when it did. We were almost upgraded.
ReplyDeleteThis comment has been removed by the author.
ReplyDelete[First post deleted for being too snarky..]
ReplyDeleteOur apologies to you and our other pre-6.2 customers for not reminding you of the existence of this bug.
The bug was first identified in June 1999. We first published information about the problem to our customers in June 2002, and the bug was fixed in GS/S version 6.2 released in November 2007. Having reported the issue to our customers and released a product version with a fix so many years ago, we sort of forgot about the problem until the customer help requests began arriving (we're up to 4 now..).
I also wish to apologize for being a bit flippant in my original post. We are aware that we are 5 years out of date with GemStone and are taking actions to correct that issue. Ideally, we would move to the 64 bit version of GemStone but that's a much more difficult transition for us because we need to change platforms from Windows to Linux.
ReplyDeleteIt's interesting that the bug was identified in 1999 and published in 2002. That was 10 years ago and virtually everyone working on the project at that time has moved elsewhere. I also appreciate that it's something that GemStone would tend to forget about - it's not something you'd put into your calendar to remind people.
My hope was that the blog post might help others locate and fix the problem.
Thanks Bill and Norm.