Efficient coding

Barlow’s suggestion, made a little over a decade after Shannon, was that the nervous system’s sensory front end should be understood as a code designed for its input statistics. Natural images are extremely redundant — neighbouring pixels are highly correlated — and transmitting that redundancy down a bandwidth-limited, metabolically expensive nerve is waste. So the early visual system should be doing something like decorrelation.

This is a rare thing in neuroscience: a normative theory. It does not describe what neurons do, it predicts what they ought to do, given the statistics of the world and a cost.

Why it earned its keep

Because the predictions came out.

Centre-surround receptive fields in retinal ganglion cells are approximately what you get if you whiten a signal with natural-image correlations. Olshausen and Field then made the sharper demonstration: train a model to represent natural image patches using as few simultaneously active units as possible, impose nothing else, and the units that emerge are localised, oriented, and bandpass — which is to say they look like the simple cells Hubel and Wiesel had recorded three decades earlier.

Nobody put edges into that model. Sparseness and natural image statistics were enough.

What the television engineers already knew

Barlow did not present the idea as new. His chapter credits Fred Attneave, who had argued in 1954 that “a major function of the perceptual machinery is to strip away some of the redundancy of stimulation”. He made the case with a version of Shannon’s guessing game for printed English: reveal a 50-by-80 picture of an ink bottle on a desk one cell at a time, with a subject guessing each cell’s colour first. By his estimate they would get through all 4,000 cells with only 15 or 20 errors, where chance would give 4,000, and the errors would fall on the contours. He said outright that this was a communications engineer’s problem: sending pictures with “the utmost economy of channel time and band width”.

The engineers were already building it. Cassius C. Cutler’s Bell Labs patent, filed in 1950, starts from the observation that correlations exist in “substantially all communication signals (for example, speech, music, or television)”, yet the typical system “employs sufficient channel capacity to transmit completely random, uncorrelated signals”. The fix, now known as differential PCM, transmits only the quantised difference between each sample and a running reconstruction of everything already sent. That reconstruction sits in a feedback loop around the quantiser, so the encoder works from exactly the history the receiver has, and the two cannot drift apart.

In 1952 B. M. Oliver published a paper under this entry’s title in the Bell System Technical Journal, and one of its two methods is the one that matters here: predict each sample from the ones before it, subtract, and send only the error — the signal with much of its correlation taken out. For television, he noted, the previous picture element alone is nearly all a linear predictor needs, because over short distances picture correlation falls off almost exponentially.

A companion paper by E. R. Kretzmer measured picture statistics and put a number on the waste: on average at least 3 bits of every 6-bit sample were redundant, enough in principle to halve the bandwidth. Peter Elias, whose Harvard thesis was on the subject, published the general theory in 1955 as “Predictive coding”.

The fly’s first synapse

Laughlin tested the idea in the fly in 1981, comparing the distribution of contrasts in natural scenes with the contrast–response curve of the fly’s large monopolar cells, the first-order interneurons that receive the photoreceptor signal. The curve approximates the cumulative distribution of the input:

$$R(c) = R_{\max} \int_{-\infty}^{c} p(c')\,dc'$$

where $p(c)$ is the probability density of contrast $c$ and $R_{\max}$ is the cell’s full response range. That shape is steep where contrasts are common and flat where they are rare, so every part of the output range is used equally often — which, for a channel with a fixed number of distinguishable levels, is the way to carry the most information. Image processing calls the same operation histogram equalisation.

The next year Srinivasan, Laughlin and Dubs carried the television method into the retina, and their reference list includes Oliver’s and Kretzmer’s papers. In their account the antagonistic surround of a receptive field takes a weighted mean of the neighbouring receptors — a statistical prediction of the centre — and subtracts it. What remains is small, so the cell’s whole dynamic range can be spent on it, and fine detail survives the noise added at later stages. It is the instrument designer’s habit of removing the offset before the high-gain stage, so that the gain can go up without clipping.

Noise is also where they went past Barlow, who had treated sensory pathways as noiseless and said so. Put it back and the best prediction depends on the signal-to-noise ratio: in dim light, when each receptor is unreliable, the surround should become weaker and more diffuse, closer to averaging than to differencing. That is what the fly’s first-order interneurons do.

Where it goes on the engineering side

Straight into signal processing, and it is one of the cleanest cases of biology being ahead:

• Predictive and transform coding in image compression are the same redundancy argument.
• Sparse representation and compressed sensing are the same objective, formalised.
• Event cameras are the argument in silicon — do not transmit what has not changed.

Ahead in one sense, and it is worth being precise about which. The retina was ahead: it was decorrelating its input long before television existed. The theory was not — predictive coding reached the eye from television, as the dates above show. The border was crossed twice.

What it leaves out

The caution worth keeping is that “efficient” needs a specified cost. Efficient in bits, in spikes, in joules, or in decoding effort are different objectives that predict different codes, and it is easy to pick whichever one matches the data you already have.

Oliver’s 1952 paper already carried a second caution: redundancy is also protection. In a predictive coder a transmission error propagates into the predictions that follow it, and with no redundancy left, he pointed out, there is no way to recognise an error at all. A nervous system built from noisy parts has the same reason to keep some.

Barlow later revised the hypothesis himself. In 2001 he judged it wrong to have over-emphasised compressive coding and economy in neuron numbers, but right to have drawn attention to the importance of redundancy — statistical structure being something the brain has to know about and exploit, not merely discard. Cortex was always the awkward case for compression: it has far more cells than the fibres that feed it, as Barlow had noted in 1961, and his own answer then was a sparse one — very few active units in a very large array.