Supermodelling

Suppose that we take a bunch of bits and break it up into two different pieces (A and B), then interpret the pair as A x 2^B, where we interpret A as a fraction between, say, -1 and 1; and B is an integer. This rather complicated way of interpreting a bunch of bits is a powerful idea — yielding numbers with good precision and a wide range of possible values. These are called floating point numbers and are really the workhorses of modern processor chips.

This is because “real numbers” (points from a continuous line of values) are incredibly useful for modelling things, and floating point numbers are for many purposes excellent approximations to real numbers. Suppose that we treat a 32-bit floating point number as a distance measured in inches. The smallest size that number can represent is way smaller than the size of a subatomic particle and the largest size is way bigger than the entire universe. And there’s a pretty decent amount of precision available too (call it 24 bits).

There are two most-common sizes for floating point numbers — 32 bits (called single precision) and 64 bits (called double precision).  The main reason for needing double precision is that 24 bits of precision is sometimes not adequate; it only lets us distinguish between about 16 million values.  That’s plenty for a lot of computation but there are two main cases where it isn’t enough:  those cases where higher precision is simply required, and cases where we have to perform arithmetic on numbers with very different scales.  As an example of the latter, suppose that we want to add 0.001 to a 32-bit floating point value 100000.0.  You’d think that the answer would be 100000.001, but it actually doesn’t work!  With 24 bits of precision, that value cannot be represented, so the answer is just 100000.0.  That is the reason that the single precision version of the Mandelbrot Set program I wrote about in another post cannot zoom in very far.

Besides distances, floats are great at representing other physical quantities (temperature, mass, etc). They are also fine for representing probabilities (although a fixed-point representation is usually better unless the need arises for truly tiny probabilities).

Just as we saw with integers, a huge amount of mathematical structure has been built up around real numbers, and approximating real numbers with a floating point representation gives access for modelling purposes to that big pile of math. To help this along, modern processors include primitive operations for addition, multiplication, and division — and also fancier things like square root, and so on. Large portions of the silicon used to build CPUs are dedicated to fast circuitry for doing these things… if one is interested in keeping a big fraction of the transistors of a CPU busy doing work, using vectors of floating point numbers (either as vectorized data types or data parallel stream processing) is surely the way to go.

That might be a frustrating conclusion if you’re interested in AGI… most peoples’ approaches to thinking about mind are not naturally expressed as vectorized floating point arithmetic. However, as a root level building block of useful modelling abstractions, floats are often excellent (which is why they exist!).

Readers might be wondering why I am plodding along on these really basic considerations, but we have to start somewhere when thinking about using computers for modelling. Our basic fundamental building blocks are bits, integers, and floating point numbers (plus a few other things like null-terminated strings which have a few specialized opcodes). What can we build from these, and what are our motivations for the next layer of complexity on top of raw bits?