Deep reinforcement learning with time-scale invariant memory

TL;DR

This paper introduces a scale-invariant memory model into deep reinforcement learning, enabling agents to learn effectively across diverse temporal scales, inspired by neuroscience principles.

Contribution

It integrates a neuroscience-inspired scale-invariant memory into deep RL, demonstrating improved adaptability over traditional recurrent architectures.

Findings

Agents with scale-invariant memory learn across wider temporal ranges

The approach outperforms LSTM-based agents in temporal generalization

Incorporating neuroscience principles enhances neural network adaptability

Abstract

The ability to estimate temporal relationships is critical for both animals and artificial agents. Cognitive science and neuroscience provide remarkable insights into behavioral and neural aspects of temporal credit assignment. In particular, scale invariance of learning dynamics, observed in behavior and supported by neural data, is one of the key principles that governs animal perception: proportional rescaling of temporal relationships does not alter the overall learning efficiency. Here we integrate a computational neuroscience model of scale invariant memory into deep reinforcement learning (RL) agents. We first provide a theoretical analysis and then demonstrate through experiments that such agents can learn robustly across a wide range of temporal scales, unlike agents built with commonly used recurrent memory architectures such as LSTM. This result illustrates that incorporating computational principles from neuroscience and cognitive science into deep neural networks can enhance adaptability to complex temporal dynamics, mirroring some of the core properties of human learning.