Monday, September 22, 2014

OpenGL ES from Smalltalk on Android

Well, here's my first rendering of an OpenGL scene from Smalltalk on the Android:

It's your standard plain colored triangle on a plain background - so shoot me.  But this image was generated by calling OpenGL primitives from my Smalltalk VM on an Android device, so it's pretty special.

I've talked about this Smalltalk before here.  Here's the current status.  I have all of the same bytecodes as VisualWorks implemented with the exception of the bytecode for thisContext. This allows me to use the VisualWorks compiler for free.

My source code is kept in a Store package in a separate namespace.  To run the code, I package all the classes and methods from that package into an image file in my own format - the Android image isn't compatible with a standard VW image.  This image is added as an asset in an Android project and sent along with a .so shared object library for the VM where it runs on the Android.

There's no Java code in the project at all.  I'm using a native activity in C and I've merged the Android event loop into the Smalltalk interpreter loop.  This allows me to handle events from the Android although I currently don't have code for that.

For debugging purposes, I can package a development image.  On startup, this image connects by a socket connection to a VW application which I wrote to provide low-level debugging and stepping instructions.

The system is still in its infancy.  There are still lots of things to develop like the garbage collector and most of the class library.  I have some basic operations but there's a lot of work needed to build up the library.  So far I don't have large integers although that's in the works.  I do support Float and Double and these classes are being used to help build the OpenGL image above.

I'll be demoing this system at Camp Smalltalk in Vancouver.  I'd be happy to answer questions about it.  It still needs a lot of work, but it's starting to produce real results.

Saturday, September 13, 2014

Camp Smalltalk in Nanaimo

Everything's booked.  My wife and I are heading to Nanaimo BC on October 3, 2014 to attend Camp Smalltalk.  I'll be demonstrating the mobile Smalltalk I'm developing - it will be a work in progress.  I may also be involved in a "Intro to Smalltalk" presentation there.  I'm heading back to Ottawa on October 6th.

If you're interested in attending, you can check out this link:

Saturday, March 1, 2014

SimTalk - Smalltalk for Mobile Devices

Ok, the secret is out. I'm busy developing a Smalltalk for Android and (later) iOS devices.  It's a new implementation of Smalltalk from the ground up. Here's what I have and what I'm planning.

I'm currently developing a pure interpreter.  Later I may consider either dynamic compilation or compilation direct to native instructions.  For now, however, the easiest way to start is with a pure bytecode interpreter.  I'm using the same bytecode set as VisualWorks.  This is mostly for convenience. It means that I can use the VisualWorks compiler to create the bytecodes for the new Smalltalk.

I don't yet have garbage collection but I have everything setup to allow me to implement garbage collection. I'll have generation scavenging for new and survivor spaces and mark and sweep for old space.

I'm developing the system using three different languages.  Java is used as a shell (for now) to start the system.  I'll also be using it to perform various primitive operations.  The core of the interpreter itself is written in C under the Android NDK.  It can call back to Java using JNI when needed.  It's the C code that loads the image from an asset and runs the Smalltalk bytecodes.

So far, I can do simple arithmetic, pushing and popping of the stack, message sends and super calls.  Any looping or branching that compiles to bytecodes also works.  I was able to write a non-recursive factorial method that worked well.

The image for the new Smalltalk is generated by VisualWorks and sent to the Android as an asset in the APK file.  When Smalltalk launches, it loads the image then sets up a socket listener and waits for a connection.  At that point, a bytecode debugger I wrote in VisualWorks connects to the image.  This debugger can provide information about the memory spaces, dump the stack, inspect objects (in a fashion), and single step through the bytecodes.

Here's what the debugger looks like so far:

This is a stack trace in the process of evaluating 3 + 4.  It's pushed the 3 and 4 onto the stack and is about to run the + method.  Remember, this is a temporary debugger for me to use at a very low level to make sure all the infrastructure is working.  Ultimately, I intend to tap into the VisualWorks debugger and direct its operations to run on the Android device.

Even though I have 95% of the VisualWorks bytecodes implemented, I'm still far away from getting the system as functional as I would like.  I need to properly implement blocks, add a lot of primitives, and develop a base class library for the new Smalltalk.  It's a lot of work but very exciting.

My current intention is to use this new Smalltalk to develop my own apps for commercial sale.  If, however, other people are interested in developing mobile apps in Smalltalk I can look into ways of licensing the application for others to use.  If this work interests you, drop me a line or leave me a comment.  I'd love to hear your thoughts.  You can also listen to the Independent Misinterpretations podcast episode 164 where I discuss the project with James.

Monday, June 24, 2013

Summer Smalltalk Training

Simberon will be offering three Webcast Smalltalk courses over the summer.

Introduction to VisualWorks Smalltalk July 22-26, 2013

Introduction to VASmalltalk Aug 19-23, 2013

Improving Object Oriented Design Aug 13-16, 2013

You don't need to travel to attend these courses.  They are all instructor-led over the Internet.  All you need is a web browser.

If you're interested in taking any of these courses, follow the links above for information and to register for the courses or email

Saturday, April 20, 2013

The Math of Music (part 3)

In the previous post, I derived the frequencies for the notes in all of the scales from Gb up to F#. In many cases, the pitches were close enough to call the same note but not quite correct. So which pitch do we choose?  We come up with a compromise called the "Equal Tempered Scale".

We know that between C and high C we need to jump from a pitch of 1 to a pitch of 2. From counting the notes in between C and high C we can see that we have 12 distinct notes.  Let's come up with a multiplier x that we can use to calculate the frequency of each of those notes.  Here's what we know:

  C = 1
  C# = 1 * x
  D = 1 * x * x
  D# = 1 * x * x * x
  C = 1 * x * x * x * x * x * x * x * x * x * x * x * x

But we know that high C = 2 so:
  x^12 = 2
  x is the 12th root of 2.

How can you calculate 12th roots?  Lets try to calculate the 4th root of 100,000,000.  Notice that there are 8 zeros.  We call 8 the log of 100,000,000 because 10 ^ 8 is 100,000,000.  If we divide 8 by 4, we get 2.  10^2 = 100 so 100 is the 4th root of 100,000,000.

We can do the same thing to calculate the 12th root of 2 by taking the log of 2 (0.30103) dividing it by 12 (0.025086) then raising 10 to that power (1.059463).  You can do the same thing with ln and exp if you're familiar with those.

So, by taking the frequency for C as 1, you can keep multiplying by 1.059463 to get each successive note until you get to high C.

Each of these pitches is close to the right note but not quite.  In the case of the black notes that could be used as sharps or flats, the pitch we calculated this way is between the sharp pitch and the flat pitch so it makes a good compromise.

All that remains is to set the actual frequencies.  By international convention, the note A is exactly 440Hz.  Given that frequency and the multiplier of 1.059463, you can calculate the pitches of every note on the piano keyboard.

The Math of Music (part 2)

In part 1, we derived the ratios of frequencies in a major scale. This gives us all the white notes on a piano but, as it turns out, they are tuned slightly differently than a piano is tuned.  The scale we derived is a "pure" scale where the tuning of all the notes is perfect and the notes make perfect harmonies.  Pianos aren't tuned to pure scales.  Each note is slightly out of tune. To discover why, we have to answer another question - why does a piano have black notes?

There are 7 notes in the C major scale (C, D, E, F, G, A, B) then the scale repeats starting back at C.  The ratios of those notes (as explained in the previous post) are shown in the table below. I've provided the fractions as well as the decimal values. Remember that we'll always divide or multiply by 2 until the answer lies in the range 1 .. 2.
D9 / 81.125
E5 / 41.25
F4 / 31.3333
G3 / 21.5
A5 / 31.6666
B15 / 81.875

This works well for playing tunes in the key of C but sometimes you want to make the pitch higher or lower. Let's choose a higher pitch.  The simplest fraction in the scale is for G (1.5 = 3/2). To create a scale in the key of G, we need to multiply all the frequencies by 1.5 like this:
G1 * 1.51.5
A1.125 * 1.51.6875
B1.25 * 1.51.875
C1.3333 * 1.5 / 21
D1.5 * 1.5 / 21.125
E1.6666 * 1.5 / 21.25
F#1.875 * 1.5 / 21.40625

Note that there are two pitches that don't match the C scale.  Our value for A is 1.6875 in the G scale but it's 1.6666 in the C scale.  It's close but still off.  The value for F, however, is really off.  In the C scale we have 1.3333 but in the G scale we're up to 1.40625. This is such a dramatic difference that you can definitely hear it and, in fact is close to halfway between the F and G in the C scale.  We'll call this note F# since it's  higher than an F but we'd like to use the letter F to have all the letters in our scale.

Now we do the same thing again.  We pick the note halfway between G (1.5) and the next G (3.0) which is D (2.25) then divide by two to get it into the 1..2 range to give D (1.125).  We multiply all the frequencies in the G scale by 1.5 (or 3/2) to create the D scale:

D1.5 * 1.5 / 21.125
E1.6875 * 1.5 / 21.2656
F#1.875 * 1.5 / 21.40625
G1 * 1.51.5
A1.125 * 1.51.6875
B1.25 * 1.5 / 21.875
C#1.40625 * 1.5 / 21.0547

Once again, our note for E is a little off from the C scale but the note for C is way off - it's about halfway between a C and a D.  We'll call it C#.

The note halfway between D and high D is A so the next key to generate is A.  I'll save you the calculations and just give the results for the remaining keys.

Notice that by the time we reach the scale of F#, we've introduced a new note called E# (1.3348) but this is so close to F (1.3333) that the notes would be virtually indistinguishable.  If we did keep going more, we'd introduce new notes that are repeats of previous notes, so we'll stop at the scale of F#.

What notes do we have?  Lets collect them up in order:

Putting the sharp notes on black keys, you get 5 black keys and 7 white keys in every octave before it repeats.

We can run the same pattern in reverse.  Starting at the scale of C (at the right side of the table below), we can divide each pitch by 1.5 (going right to left). Now, we create new notes that we'll call Flats.  If you check the frequencies, though, the flats are almost the same frequencies as the corresponding sharps - Bb (1.7777) is the same as almost A# (1.7797), Eb (1.1852) is almost the same as D# (1.1865). 

We're left with a problem.  If we want to play in any scale with perfect tuning, we need an infinite number of keys.  Fortunately, some of the pitches end up being close enough to other notes that we can use the other note instead.

So how do we come up with a compromise?  That's the subject of part 3 in this series.

Friday, April 19, 2013

The Math of Music (Part 1)

I've always been interested in the mathematical theory behind music.  When I started programming computers in 1977, I wrote programs to play music so I needed to calculate the note frequencies.  I was never able to understand, however, why the piano keys are the way they are.  Why is there no E# or Fb note? Why no B# or Cb?  Why is the same key sometimes called F# and sometimes called Gb?

I've since learned how this works and it's all simple math.  In this post, I'll try to explain it.

Let's start with Middle C on the piano keyboard.  I really don't care what frequency it is.  Musical notes are all about ratios, so let's just call the frequency c.  What notes sound good when you play them at the same time as the middle C note?  Well, a note that's twice the frequency sounds good.  For every cycle of the C note, the note at 2c makes two cycles. This blends nicely with the C and makes a nice sound.  This note is high C - the next C up on the keyboard.

Another note that sounds good along with middle C is at 1/2 c. This is the next C down on the keyboard. To go up and down by octaves, we double (to go up) or half (to go down) the frequency of a note.  From a scale perspective, I'm going to consider all C notes equivalent.

Ok, what note is halfway between c and 2c?  Obviously it's 1.5c. (Later we'll call this G but we'll get to that.) This note makes three cycles for every two cycles of the C note.  It sounds very nice played along with a C so we'll include it.

The next note that sounds good is halfway between 1c and 1.5c.  That's 1.25c. (Later we'll call this E.) This note makes 5 cycles for every 4 cycles of the C note.

If you play these three notes all at once you get what is called a major chord.

       1c    1.25c   1.5c          2c     Major Chord

Major chords sound very nice.  Let's then build a major chord starting at the 1.5c note.  The frequencies would be (I'll drop the "c" notation for simplicity):

    1.5 * 1 = 1.5
    1.5 * 1.25 = 1.875
    1.5 * 1.5 = 2.25

       1     1.25    1.5           2     C Major Chord
                     1.5   1.875     2.25
                      |-------|-------|  G Major Chord

We'll later call these notes G, B and D but we'll get to that.

The note at 2.25 is musically equivalent to the same note an octave down so let's divide the frequency by 2 to put it into the range 1..2

    1.5 * 1 = 1.5
    1.5 * 1.25 = 1.875
    1.5 * 1.5 / 2 = 1.125

       1     1.25     1.5          2     C Major Chord
         1.125        1.5   1.875
           |-----------|-------|         G Major Chord

To make this major chord, we picked the middle note (1.5) and made a scale from it.  Let's do the opposite - let's make a chord where the C note is the middle note.  That means we get these frequencies:

    1 / 1.5 = 0.6666...
    1 / 1.25 = 0.8333...

                 1     1.25    1.5           2     C Major Chord
 0.6666  0.8333  1        1.3333

We'll later label these notes as F, A and C.

We've structured this so that the C note at 1 is halfway between 1.3333 and 0.6666.

Since 0.666 and 0.833 are less than 1, we can double them to make equivalent notes in the range 1..2:

    1 / 1.5 * 2 = 1.3333...
    1 / 1.25 * 2 = 1.6666...

        1     1.25    1.5           2     C Major Chord
        1       1.3333  1.6666

Ok, now let's take all the notes we have between 1 and 2 and arrange them by increasing frequency:


Now, we can finally give these all names based on sequential letters:

   C 1
   D 1.125
   E 1.25
   F 1.3333
   G 1.5
   A 1.6666
   B 1.875
   C 2

The notes C E G form one major chord (C), the notes G B D form another major chord (G) and finally the notes F A C form a third major chord (F).  All combined together, we'll call this the scale of C.

In the next article, I'll start talking about sharps and flats.