A Computational Theory of Learning Flexible Reward-Seeking Behavior with Place Cells

TL;DR

This paper introduces a biologically plausible computational model using place cells and synaptic learning rules that enables flexible reward-seeking behavior in a virtual rat, outperforming traditional reinforcement learning methods.

Contribution

The model combines place cell dynamics, Hebbian learning, and Markov chain replay to achieve flexible, biologically plausible reward-seeking behavior in complex environments.

Findings

The virtual rat learns reward locations more efficiently.

The model demonstrates greater behavioral flexibility.

Outperforms deep Q-network in complex maze tasks.

Abstract

An important open question in computational neuroscience is how various spatially tuned neurons, such as place cells, are used to support the learning of reward-seeking behavior of an animal. Existing computational models either lack biological plausibility or fall short of behavioral flexibility when environments change. In this paper, we propose a computational theory that achieves behavioral flexibility with better biological plausibility. We first train a mixture of Gaussian distributions to model the ensemble of firing fields of place cells. Then we propose a Hebbian-like rule to learn the synaptic strength matrix among place cells. This matrix is interpreted as the transition rate matrix of a continuous time Markov chain to generate the sequential replay of place cells. During replay, the synaptic strengths from place cells to medium spiny neurons (MSN) are learned by a temporal-difference like rule to store place-reward associations. After replay, the activation of MSN will ramp up when an animal approaches the rewarding place, so the animal can move along the direction where the MSN activation is increasing to find the rewarding place. We implement our theory into a high-fidelity virtual rat in the MuJoCo physics simulator. In a complex maze, the rat shows significantly better learning efficiency and behavioral flexibility than a rat that implements a neuroscience-inspired reinforcement learning algorithm, deep Q-network.