The silicon retina

A conventional image sensor answers a question nobody asked: what is the brightness of every pixel, right now, thirty times a second, whether or not anything moved. The retina does not work this way, and Mahowald and Mead’s silicon retina was an argument that the difference matters for engineering and not only for biology.

What it is

An analog CMOS chip that Misha Mahowald, a graduate student at Caltech, and Carver Mead described in 1988. It copies the first stage of a vertebrate retina, where photoreceptors, horizontal cells and bipolar cells meet, with a circuit in every pixel for each.

A photoreceptor that takes a logarithm. A vertical bipolar transistor, a by-product of the CMOS process, turns light into current, and two diode-connected MOS transistors in series turn the current into a voltage. Below threshold a MOS transistor’s current is exponential in its gate voltage, so the voltage is logarithmic in the light, over four to five decades. In logarithms a ratio becomes a difference: the voltage between two pixels measures the contrast between two points of the image, whatever the illumination.

A resistive network for the horizontal cells, which in many species are joined by gap junctions into a continuous sheet. Resistors made from transistors, since the process offers none of high enough value, tie each pixel to its six neighbours in a hexagonal grid, and each photoreceptor drives its own node. The voltage at a node is an average of the receptors around it, weighted less with distance, and the grid’s capacitance makes it an average over time as well.

An amplifier for the bipolar cell, which reports the difference between the receptor and the network at that point: centre minus surround.

So a pixel responds most where the image changes in space or in time — in Mead’s later description, the output enhances the image’s first temporal and second spatial derivatives. Their 48 × 48 array fitted on a quarter of a square centimetre and drew 100 µW.

The computation ran continuously, in analog. The readout did not: shift registers along two edges of the chip selected one pixel at a time, either holding on one, like an electrode in a single cell, or scanning the array frame by frame onto a television monitor.

Reporting only change

The borrowing was lateral inhibition, the subtraction of a local average, and the computation is the one Ernst Mach had inferred from perception in 1868: a point looks brighter or darker according to how its light compares with a distance-weighted average of its neighbours’. The chip duly showed Mach bands.

Mead’s engineering reading was that the main job is automatic gain control. The local average sets each pixel’s operating point, so the output can use its full gain whatever the overall light level, and because the reference is local, detail survives in the bright and the dark parts of one scene — which, Mead and Mahowald noted, would overwhelm a television camera adapting only globally.

He also framed it as prediction. The retina assumes that brightness varies at most linearly across the image and not at all in time, and passes on only where the assumption fails — the idea Srinivasan, Laughlin and Dubs had called predictive coding in 1982. Most pixels, Mahowald observed, have almost nothing to say. Change, in 1988, meant departure from a neighbourhood’s recent average, in space or in time; the event camera would keep only the temporal half.

From a scan to a stream

The scan threw much of that saving away. Every pixel was sampled every frame whether or not it had anything to report, and a change waited on average half a frame to be read out.

Kwabena Boahen credits the answer to Mahowald and to Massimo Sivilotti, whose 1991 thesis designed the arbiter at its heart; Mahowald’s 1992 thesis named it the address-event representation. Each pixel becomes a spiking neuron, an integrate-and-fire circuit that charges a capacitor to threshold and resets. When it fires, the arbiter grants it a shared digital bus and its address goes out as an event about a microsecond long. Arrival time carries the timing, so the address is all that need be sent: a thousand fibres, if only one fires at a time, can share about eleven wires.

Her demonstration retina, 64 × 64 pixels, fired on increases in light, like an on-transient ganglion cell, and copied a time-derivative image onto a receiver chip. She estimated that a typical image drives about a quarter of a retina’s pixels above their spontaneous rate, 86 events a second per pixel on average, and calculated that no queue would form. By 2000 Boahen could call the scheme the leading candidate for linking neuromorphic chips.

Later chips went back to the biology. Kareem Zaghloul and Boahen’s 2004 retina modelled the inner retina too, with amacrine-cell feedback for contrast gain control, and sent the spikes of 3,600 ganglion cells of four types — ON and OFF, sustained and transient — off chip as address-events, from 3.5 × 3.3 mm of silicon drawing 62.7 mW.

The event camera took the other branch: one kind of output, change in time, pixel by pixel. Patrick Lichtsteiner, Christoph Posch and Tobi Delbrück described its first practical form, the dynamic vision sensor, in 2008; a survey Delbrück co-wrote traces it to a photoreceptor array he and Mead described in 1991. Each pixel contains a photoreceptor with logarithmic response and a circuit that compares the log intensity with its value at the pixel’s last event. A pixel that sees no change says nothing. A pixel whose log-intensity shifts past a threshold emits an event, immediately and asynchronously, and resets its reference.

There is no frame, and no clock defining one. The output is a stream of events in continuous time.

Why it matters

Three consequences fall out of that design, all of them things frame-based sensors struggle with:

• Dynamic range. Because adaptation is local and logarithmic, the sensor handles a scene with bright sky and deep shadow simultaneously — more than 120 dB in the 2008 sensor, against about 60 dB for a good frame-based camera, which must choose an exposure and lose one end.
• Latency. An event is emitted when the change happens, not when the next frame is read out. Microseconds instead of milliseconds.
• Bandwidth and power. A static scene costs almost nothing to report. Data rate scales with activity in the world, not with pixel count and frame rate.

That last point is sparse coding doing exactly what it does in biology: spend the power budget on the channels that carry information and leave the rest quiet. The retina did not arrive at this because change detection is elegant; it arrived there because the optic nerve is a bandwidth bottleneck and metabolism is expensive. By Boahen’s estimate, a camera sampling at the Nyquist rate would need about 40 Gb/s to match the eye’s photopic range, spatial and temporal resolution and field of view; at about 2 bits a spike, the optic nerve’s million axons carry some 40 Mb/s.

The engineering lesson generalises past vision, and this is the part worth taking seriously: the constraint that shaped the biology — limited channel capacity, limited energy — is the same constraint on an embedded sensor. Where the constraints match, the biological solution is not a metaphor. It is a design.

What it cost

Mismatch, first. Logarithmic compression left the variations within an evenly lit scene about as small as the offsets between nominally identical transistors, so under uniform light the 1988 chip’s output was a mottled pattern of its transistors rather than of the scene. Mead’s adaptive retina of 1989 trimmed each pixel’s offset with charge on a floating gate, set by a negative feedback loop that ran only while the chip was bathed in ultraviolet light.

Spiking made offsets costlier. Mahowald’s address-event pixels integrated them along with the signal, so much of the bus carried offset rather than image, and she judged the pixel not sensitive enough to make a practical imager. The 2008 sensor held the variation of its event threshold to 2.1% across the array, but noise and mismatch still limit the smallest change an event camera can usefully report, typically set between 10% and 50%.

The economy is conditional, too. In Mahowald’s queueing model, address-events beat the scan’s timing by a wide margin at low event rates and fell behind quickly once events arrived faster than the bus could serve them; a busy scene can still saturate an event camera’s bus. And the 15 µs latency of 2008 was a bright-light best case.

Reporting only change means never reporting what stays the same. Point a still sensor at a still scene and, noise apart, it reports nothing, which is why later sensors added a conventional intensity readout beside the events.