Representation learning with reward prediction errors

TL;DR

This paper extends the Reward Prediction Error hypothesis by proposing that dopamine signals also support learning of state representations, explaining diverse functions like time perception, motor responses, and spatial cognition.

Contribution

It introduces a novel view that dopamine prediction errors help acquire adaptive state representations, broadening the understanding of dopamine's role beyond reward learning.

Findings

Simulations show dopamine prediction errors support temporal and spatial representations.

The model accounts for dopamine's role in motor responses and categorization.

Extends the Reward Prediction Error hypothesis to include representation learning.

Abstract

The Reward Prediction Error hypothesis proposes that phasic activity in the midbrain dopaminergic system reflects prediction errors needed for learning in reinforcement learning. Besides the well-documented association between dopamine and reward processing, dopamine is implicated in a variety of functions without a clear relationship to reward prediction error. Fluctuations in dopamine levels influence the subjective perception of time, dopamine bursts precede the generation of motor responses, and the dopaminergic system innervates regions of the brain, including hippocampus and areas in prefrontal cortex, whose function is not uniquely tied to reward. In this manuscript, we propose that a common theme linking these functions is representation, and that prediction errors signaled by the dopamine system, in addition to driving associative learning, can also support the acquisition of adaptive state representations. In a series of simulations, we show how this extension can account for the role of dopamine in temporal and spatial representation, motor response, and abstract categorization tasks. By extending the role of dopamine signals to learning state representations, we resolve a critical challenge to the Reward Prediction Error hypothesis of dopamine function.