Theta sequences as eligibility traces: a biological solution to credit assignment

TL;DR

This paper proposes that hippocampal theta sequences serve as biological eligibility traces, enabling credit assignment in reinforcement learning without long-term memory traces, by compressing behavior during theta oscillations.

Contribution

It introduces a model showing theta sequences can function as eligibility traces, providing a biologically plausible solution to credit assignment problems.

Findings

Theta sequences effectively extend neuronal memory traces.

Model demonstrates bootstrap-free credit assignment in hippocampal activity.

Simulation results align with reinforcement learning eligibility trace mechanisms.

Abstract

Credit assignment problems, for example policy evaluation in RL, often require bootstrapping prediction errors through preceding states \textit{or} maintaining temporally extended memory traces; solutions which are unfavourable or implausible for biological networks of neurons. We propose theta sequences -- chains of neural activity during theta oscillations in the hippocampus, thought to represent rapid playthroughs of awake behaviour -- as a solution. By analysing and simulating a model for theta sequences we show they compress behaviour such that existing but short $\mathsf{O}(10)$ ms neuronal memory traces are effectively extended allowing for bootstrap-free credit assignment without long memory traces, equivalent to the use of eligibility traces in TD($\lambda$).