Emergent time-keeping mechanisms in a deep reinforcement learning agent performing an interval timing task

TL;DR

This study demonstrates that a deep reinforcement learning agent can develop internal oscillatory mechanisms resembling biological time-keeping, specifically paralleling the Striatal Beat Frequency model, to perform interval timing tasks.

Contribution

It reveals emergent oscillatory neural activations in a DRL agent that mimic biological timing mechanisms, advancing understanding of temporal processing in artificial and biological systems.

Findings

Agent successfully performed interval timing with internal oscillations.

Oscillatory neurons influenced action timing, mirroring biological patterns.

Time-keeping persisted across different environmental conditions.

Abstract

Drawing parallels between Deep Artificial Neural Networks (DNNs) and biological systems can aid in understanding complex biological mechanisms that are difficult to disentangle. Temporal processing, an extensively researched topic, is one such example that lacks a coherent understanding of its underlying mechanisms. In this study, we investigate temporal processing in a Deep Reinforcement Learning (DRL) agent performing an interval timing task and explore potential biological counterparts to its emergent behavior. The agent was successfully trained to perform a duration production task, which involved marking successive occurrences of a target interval while viewing a video sequence. Analysis of the agent's internal states revealed oscillatory neural activations, a ubiquitous pattern in biological systems. Interestingly, the agent's actions were predominantly influenced by neurons exhibiting these oscillations with high amplitudes and frequencies corresponding to the target interval. Parallels are drawn between the agent's time-keeping strategy and the Striatal Beat Frequency (SBF) model, a biologically plausible model of interval timing. Furthermore, the agent maintained its oscillatory representations and task performance when tested on different video sequences (including a blank video). Thus, once learned, the agent internalized its time-keeping mechanism and showed minimal reliance on its environment to perform the timing task. A hypothesis about the resemblance between this emergent behavior and certain aspects of the evolution of biological processes like circadian rhythms, has been discussed. This study aims to contribute to recent research efforts of utilizing DNNs to understand biological systems, with a particular emphasis on temporal processing.