Hippocampal representations emerge when training recurrent neural networks on a memory dependent maze navigation task

TL;DR

This study demonstrates that recurrent neural networks trained on maze navigation tasks develop hippocampus-like representations, including place cells and anticipatory activity, through combined predictive and reinforcement learning.

Contribution

It shows how integrating predictive learning with Q-learning in RNNs leads to hippocampal-like spatial and task-specific neural representations, paralleling experimental observations.

Findings

Attractor landscapes form in network representations after environment structure learning.

Reward-dependent training causes neural activity to sweep forward, predicting future paths.

Choice and cue-selective neurons emerge, mirroring hippocampal activity in rodents.

Abstract

Can neural networks learn goal-directed behaviour using similar strategies to the brain, by combining the relationships between the current state of the organism and the consequences of future actions? Recent work has shown that recurrent neural networks trained on goal based tasks can develop representations resembling those found in the brain, entorhinal cortex grid cells, for instance. Here we explore the evolution of the dynamics of their internal representations and compare this with experimental data. We observe that once a recurrent network is trained to learn the structure of its environment solely based on sensory prediction, an attractor based landscape forms in the network's representation, which parallels hippocampal place cells in structure and function. Next, we extend the predictive objective to include Q-learning for a reward task, where rewarding actions are dependent on delayed cue modulation. Mirroring experimental findings in hippocampus recordings in rodents performing the same task, this training paradigm causes nonlocal neural activity to sweep forward in space at decision points, anticipating the future path to a rewarded location. Moreover, prevalent choice and cue-selective neurons form in this network, again recapitulating experimental findings. Together, these results indicate that combining predictive, unsupervised learning of the structure of an environment with reinforcement learning can help understand the formation of hippocampus-like representations containing both spatial and task-relevant information.