Reward prediction error

Midbrain dopamine neurons fire when a monkey receives unexpected juice. Train the monkey so a light predicts the juice, and the firing moves to the light; the juice itself, now expected, produces nothing. Omit the juice after the light and the cells go below baseline at exactly the moment it was due.

That is not a reward signal. It is an error signal — reward minus prediction, signed.

The crossing

This is the clearest case on the site of a formalism arriving first and turning out to be what the biology was already doing.

The problem is one of delayed feedback, familiar to anyone who has tuned a controller: if the consequence of an action arrives long after the action, how do you assign credit to the right earlier state? Temporal-difference methods answer it the way a control engineer would — do not wait for the outcome, correct on the discrepancy between successive predictions:

$$\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$$

Here $V(s_t)$ is the predicted sum of reward from time step $t$ onwards, given the situation $s_t$; $r_t$ is the reward actually received; and $\gamma$, between 0 and 1, discounts reward the further off it lies. The predictions are corrected by $\delta_t$, and when they are right it averages zero.

Schultz, Dayan and Montague pointed out that $\delta_t$ — a quantity devised so that a learner could correct its predictions step by step, before the outcome arrives — has the same signature as the dopamine response, including the shift to the predictive cue and the dip below baseline on omission. The formalism was not built to explain dopamine. It already existed, and the neurons matched it.

That is a stronger claim than an analogy. The dip on omission is the part that convinces: a system merely reporting reward has no reason to go negative when nothing happens. A system reporting error has no choice.

The dip is also on time, and in the model that takes a clock. There each cue becomes a row of signals, one per time step after its onset, each with a weight that moves by its own signal times $\delta_t$: a tapped delay line with an adaptive weight on every tap, trained by an error-times-input rule of the Widrow–Hoff kind. The taps learn a prediction that stands from the cue until the juice is due, then drops away. Juice that arrives pays off the drop; juice that does not leaves it showing as a negative $\delta_t$ at that delay and no other. In the recordings the dip came more than a second after the cue, with no external stimulus in between to mark the time.

Borrowed: an error term from control

In 1983, in the IEEE Transactions on Systems, Man, and Cybernetics, Andrew Barto, Richard Sutton and Charles Anderson set a learning controller to balance a pole hinged to a cart. It had no model of the dynamics and one piece of feedback: −1 when the pole tipped past 12° or the cart hit either end of its track, zero the rest of the time. To make that rare, late signal usable they added an adaptive critic, which learned to predict failure from the state of cart and pole and passed the controller, at every 20 ms step of the simulation,

$$\hat{r}(t) = r(t) + \gamma p(t) - p(t-1)$$

where $p$ is the critic’s prediction. That is the same quantity as $\delta_t$ — reward, plus the new prediction discounted, minus the old one — with its time indices written differently. Moving into a state the critic rated more dangerous was punished as failure itself had been, moving to a safer one was rewarded, and, in the authors’ words, “a fully predicted failure generates no punishment.” Read juice for failure and that is the dopamine result, in a paper about balancing a pole. With the critic, runs tended to solve the problem within 100 failures; their version of Michie and Chambers’ earlier BOXES controller tended not to.

Part of the lineage is adaptive filtering and control. The authors likened their rule to Widrow and Hoff’s ADALINE rule, and related $\gamma$ to the discount factor of Ian Witten’s 1977 adaptive optimal controller. Sutton’s 1988 paper then treated temporal-difference methods as a class in their own right for the first time and proved that they converge in special cases. Its first theorem shows that one member of the family makes the same weight changes over a sequence as the Widrow–Hoff least-mean-squares rule, computed step by step instead of after the outcome is known.

That is the counterpart: a prediction, and the signed difference that corrects it, worked out and tested as a problem in adaptive control. One qualification belongs beside it. Sutton and Barto had been modelling animal conditioning with the same kind of element since at least 1981, as an extension of the Rescorla–Wagner model that also accounts for timing within a trial, and the 1997 paper says the algorithm was originally inspired by data on how animals learn to predict. It was, though, shaped by the animal learning that dopamine turned out to help drive, which makes the match less of a coincidence than it first looks.

The recordings, and the reading

The recordings came from Wolfram Schultz’s laboratory in Fribourg: single dopamine neurons in monkeys learning to work for juice. In a 1993 study with Apicella and Ljungberg, 25% of 76 neurons responded to the juice while a task was being learned, against 9% of 163 once it was established; responses to the lights that predicted the juice persisted. The authors read this as attention and motivation, and took the lack of sustained firing in the task’s delay period to mean that the cells did not encode an expectation of reward.

Theorists supplied the other reading. After several conference papers, Read Montague, Peter Dayan and Terrence Sejnowski set it out in full in 1996: fluctuations above and below baseline as errors in a prediction of future reward, broadcast to cortical and subcortical targets, with a Hebbian learning rule — presynaptic activity correlated with the broadcast error — taken from temporal-difference methods. On that reading the missing sustained firing is just what the theory expects. Their papers credit similar readings from others at about the same time, among them Houk, Adams and Barto’s model of the basal ganglia, and the 1997 paper in Science brought Schultz’s data and the model into one argument.

Injecting the error

A matching signature is still a correlation. The engineering test of an error signal is to inject one and see whether the loop adapts, and optogenetics made a brief, cell-specific injection possible. Steinberg and colleagues (2013) used blocking: a cue added alongside one that already predicts a reward is learned only weakly, because the reward it arrives with is no surprise. Rats learned that a sound predicted sucrose, then had a flashing light added to the sound, and during that phase their dopamine neurons were driven with 1 s trains of 5 ms laser pulses at 20 Hz, timed to the sucrose. Tested alone the next day, the light drew more reward-seeking on its first presentation than in either control group; the same trains delivered between trials left it blocked.

Chang and colleagues (2016) supplied the other sign, which mattered because a positive effect alone could be read as novelty or salience: brief optogenetic pauses in firing at the time of reward stood in for a negative error. And Eshel and colleagues (2015) found part of the comparator. In mice, dopamine neurons subtract expected from received reward, and neighbouring GABA neurons in the ventral tegmental area, which inhibit them when reward is expected, are one source of the subtraction.

Caveats worth stating

The story is cleaner in review articles than in the data. Dopamine also responds to novelty, salience, and stimulus intensity in ways a pure scalar error term does not predict; there are multiple dopamine populations with different tuning; and the distributional picture that has emerged more recently suggests neurons encode a distribution over rewards rather than one expected value. The prediction-error account remains the best organising idea in the area, which is not the same as being complete.

The distributional result is Dabney and colleagues’ (2020), and it reads naturally as a circuit. An error channel that weighs positive errors more heavily than negative ones settles not at the mean of the rewards it sees but above it, where its weighted errors balance; one that weighs negative errors more settles below. A bank of channels with different asymmetries spans the distribution. In optogenetically identified dopamine neurons of mice given a liquid reward of one of seven sizes, drawn at random on each trial, the same 5 µL drop — measured 200–600 ms after delivery, to skip an initial burst that every reward size produced — left 13 of 40 cells significantly above baseline and 10 significantly below, and cells that scaled positive errors more steeply had higher reversal points — the reward size at which a cell’s response changes sign — as the scheme requires. The idea came from artificial-intelligence research; the evidence came from single neurons.

The delay line that times the dip is an assumption too. The 1997 paper concedes that almost nothing was known about how the brain holds a representation of a stimulus over long delays. The recordings show that something keeps time from the cue; the model shows what would suffice, not what is there.

The reason this sits close to my own work: an error signal is only worth having if something can act on it. A controller needs an actuator, and for the nervous system that means stimulation you can shape and time rather than the fixed pulse trains a conventional implant delivers open-loop.