Spatial-Aware Decision-Making with Ring Attractors in Reinforcement Learning Systems

TL;DR

This paper introduces the use of biologically inspired ring attractors in reinforcement learning to enhance spatial decision-making, leading to significant performance improvements in Atari benchmarks.

Contribution

It presents a novel integration of ring attractors into deep reinforcement learning, improving learning speed, accuracy, and stability in action selection.

Findings

Achieved 53% performance increase on Atari 100k benchmark.

Demonstrated improved stability in action selection during exploration.

Validated the effectiveness of ring attractors in neural network-based RL.

Abstract

Ring attractors, mathematical models inspired by neural circuit dynamics, provide a biologically plausible mechanism to improve learning speed and accuracy in Reinforcement Learning (RL). Serving as specialized brain-inspired structures that encode spatial information and uncertainty, ring attractors explicitly encode the action space, facilitate the organization of neural activity, and enable the distribution of spatial representations across the neural network in the context of Deep Reinforcement Learning (DRL). These structures also provide temporal filtering that stabilizes action selection during exploration, for example, by preserving the continuity between rotation angles in robotic control or adjacency between tactical moves in game-like environments. The application of ring attractors in the action selection process involves mapping actions to specific locations on the ring and decoding the selected action based on neural activity. We investigate the application of ring attractors by both building an exogenous model and integrating them as part of DRL agents. Our approach significantly improves state-of-the-art performance on the Atari 100k benchmark, achieving a 53% increase in performance over selected baselines.