Neuromorphic computing

Mead’s observation was quantitative and uncomfortable: the nervous system was performing operations at a small fraction of the energy per operation that the digital hardware of the day required — a gap of orders of magnitude, far too large to be explained by process technology. It had to come from the style of computation.

Where the energy went

The figures, in the 1990 paper in which Mead called such systems neuromorphic, were round and explicit. A typical microprocessor did about 10 million operations a second on about a watt: $10^{-7}$ J per operation. The brain, counting each arrival of a nerve pulse at a synapse as an operation, managed about $10^{16}$ a second on a few watts, roughly $10^{-16}$ J apiece — by Mead’s reckoning a billion times more efficient than the digital technology of the day, and ten million times more than the best digital technology anyone could imagine.

The transistor was not the problem. Charging the gate of a minimum-size device took about $10^{-13}$ J, heading for $10^{-15}$ J within a decade. The missing factor of a million lay between one transistor and one digital operation: about 100 because a node’s capacitance is mostly wire rather than gate, and about 10,000 because an operation switches some 10,000 transistors. More local algorithms could win back the first factor, digital or analog. The second was the price of building arithmetic out of AND and OR gates instead of letting the device physics do it. His adaptive silicon retina, about $10^5$ devices doing the equivalent of $10^8$ operations a second on a milliwatt, came to $10^{-11}$ J per operation: four orders of magnitude ahead of digital in the same process, five behind the brain.

What it is

Two commitments, which are separable and often confused:

• Analog, subthreshold operation. Run transistors in the regime where drain current depends exponentially on gate voltage. The device physics then performs the arithmetic directly — an exponential, a multiplication, a sum of currents at a node — instead of digital logic simulating it. Enormously efficient, and inherently imprecise.
• Spikes and events. Represent values in the timing of asynchronous events rather than in clocked words. No global clock, no activity where nothing is happening.

The first rests on one device law. Below threshold, a MOS transistor’s current is carried by diffusion over an energy barrier that the gate voltage raises and lowers, and because the carriers’ energies are Boltzmann-distributed the current is exponential in that voltage:

$$I = I_0 \, e^{\kappa V_g / U_T}$$

Here $V_g$ is the gate voltage, with source and substrate tied together and the drain several $U_T$ above them; $I_0$ is a leakage current that scales with the transistor’s width-to-length ratio; $U_T = kT/q$ is the thermal voltage, about 25 mV at room temperature ($k$ Boltzmann’s constant, $T$ absolute temperature, $q$ the electron’s charge); and $\kappa$, the subthreshold slope factor, is the fraction of the gate voltage that acts on the channel. The currents run from femtoamps to nanoamps. Mead plotted the curve beside the currents through populations of ion channels, exponential for the same reason. Kirchhoff’s current law then adds currents at a node for nothing, the node’s capacitance integrates them, and a chain of exponentials turns a sum of voltages into a product of currents. In Sarpeshkar’s later count, adding two 8-bit numbers takes one wire in analog and about 240 transistors of static CMOS logic; multiplying them, 4 to 8 transistors against about 3,000.

The regime itself was not new. In 1977 Vittoz and Fellrath, at the Centre Electronique Horloger in Neuchâtel, gave circuit designers a two-parameter model of weak inversion and micropower circuits built on it, among them a 32 kHz quartz oscillator drawing under 0.1 µW. Mead’s step was to treat the exponential as a computing primitive shared with the nerve membrane, not only as a way to run circuits on microwatts. His 1989 book, Analog VLSI and Neural Systems, applied it to the retina and the cochlea, and its axon-hillock circuit is a silicon integrate-and-fire neuron. In 1991 Mahowald and Douglas built the ionic conductances of a Hodgkin–Huxley-style neuron from transconductance amplifiers running below threshold.

What was borrowed from the spike

What crossed under the label on the far side of the axis, the brain’s spiking, event-driven efficiency, was not the action potential’s shape or its ionic machinery — the analog branch borrowed those separately — but a way of communicating, and a way of paying for it.

A spike’s height and width are fixed, so all it carries is which cell fired and when. Put the address of the neuron that fired on a bus at the moment it fires, and a receiver working in real time learns everything the pulse could have told it — “time represents itself”, in Boahen’s phrase. Mahowald and Sivilotti proposed this address-event representation at Caltech, in theses of 1991 and 1992, to answer a wiring problem: a logic gate drives about ten others where a neuron reaches several thousand, and a chip package has few pins. A neuron signals at hundreds of hertz and a digital bus at tens of megahertz, and that five-decade gap lets a handful of shared wires stand in for thousands of dedicated ones.

The payment is the other half. Spikes are expensive, and Lennie estimated in 2003 that their cost may hold human cortex to fewer than 1 per cent of neurons substantially active at once — sparse coding forced by an energy budget. Event-driven hardware copies the shape of that bill, no spike and no traffic, and it copies the condition too. Boahen worked out the break-even for a 64 × 64 array. A scan sends one bit per neuron per sample, the address implied by its place in the sequence; an event carries twelve bits to name its neuron. Even if neurons with static input report forty times less often than active ones, events win only while fewer than about 6 per cent of the neurons are active. The borrowed efficiency holds only while activity stays sparse — the condition the brain imposes on itself, and the one an event camera gets from a still scene.

Where the two commitments went

The large systems that followed kept the events and divided over the analog. Neurogrid, from Boahen’s group at Stanford, kept both, the analog for its efficiency — subthreshold circuits for everything but the axonal arbors, which are digital address-event routing — and in 2014 simulated a million neurons in real time on a board drawing three watts. The Heidelberg system that became BrainScaleS (Schemmel and colleagues, 2010) built the wafer-scale integration Mead had argued low-power analog would allow, 384 chips wired together on an uncut 8-inch wafer, and spent the physics on speed: its small capacitances run the dynamics about 10,000 times faster than biology.

The rest went digital and kept only the events. SpiNNaker, at Manchester, runs neuron models in software on 18 ordinary ARM cores per chip, each halted until an interrupt — an arriving spike packet, or its millisecond timer — wakes it. IBM’s TrueNorth (2014) fits a million neurons and 256 million synapses into 5.4 billion transistors, drew 63 mW on 400 × 240-pixel video at 30 frames a second, and was designed so that its active power follows firing activity. Intel’s Loihi (2018) adds on-chip learning confined to information local to each synapse, as in Hebbian learning, and replaces the global clock with handshakes between cores.

Counted per synaptic event, TrueNorth and Loihi come to about 25 pJ — 26 pJ for TrueNorth with neurons firing at 20 Hz onto 128 active synapses each, 23.6 pJ at best in Loihi’s pre-silicon figures. That is the order of Mead’s analog retina in 1990, and still some five orders of magnitude from his figure for the brain.

Why it matters

The efficiency argument holds up: sparse, event-driven, analog computation genuinely avoids costs that clocked digital designs pay unavoidably. Moving a word across a chip and back to memory dominates the energy budget of a conventional digital accelerator, and an architecture with local state and no clock simply does not incur that cost.

The honest difficulty is that precision and programmability were what digital computing bought with all that energy, and neuromorphic designs give both back. Analog devices drift, mismatch between nominally identical transistors is substantial, and there is no equivalent of a compiler for a substrate whose behaviour varies across the die. Every serious neuromorphic project has had to answer how you program the thing, and the answers remain less satisfying than the efficiency numbers.

Precision has a price list. In Sarpeshkar’s 1998 analysis the cost of analog computation grows roughly as a power of its signal-to-noise ratio and the cost of digital only as its logarithm, so analog wins at low precision and loses at high; for many common computations in the CMOS of the day the crossover fell near 8 bits. An analog signal is also never restored between stages, so noise accumulates down a cascade.

Mismatch has had three answers. Mead’s was the brain’s: adaptation. In his adaptive retina, ultraviolet light let each pixel’s floating gate trade charge with its output, a negative feedback loop that drove every output towards the same level and so cancelled the transistors’ offsets — the same mechanism, he argued, that lets a system learn a model of its input. BrainScaleS calibrates instead, and calibration runs out: there are too many synapses to store calibration data for each, and its floating-gate parameter memories reproduce their values only approximately at each rewrite, a run-to-run variation (assumed under 30 per cent of a synaptic weight) that no calibration removes. TrueNorth removed the variation. It is digital, steps its neurons on a global 1 kHz clock, and runs a network spike for spike the same as its software simulator. Determinism is what a compiler needs, and TrueNorth bought it by giving up the analog half of Mead’s argument.

This is worth stating plainly because the field’s rhetoric has often outrun its results. The energy argument is sound. The programming story is unfinished — which makes it interesting rather than settled.