Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learning

TL;DR

This paper introduces the Stable Hadamard Memory, a new memory model for reinforcement learning that dynamically manages past experiences to improve decision-making in partially observable environments, outperforming existing methods.

Contribution

The paper presents the Stable Hadamard Memory, a novel, efficient memory model that adaptively updates and stabilizes memory in reinforcement learning agents, addressing limitations of prior models.

Findings

Outperforms state-of-the-art memory methods on challenging benchmarks

Enhances memory capacity while reducing numerical and learning challenges

Effectively handles long-term and evolving contexts in RL environments

Abstract

Effective decision-making in partially observable environments demands robust memory management. Despite their success in supervised learning, current deep-learning memory models struggle in reinforcement learning environments that are partially observable and long-term. They fail to efficiently capture relevant past information, adapt flexibly to changing observations, and maintain stable updates over long episodes. We theoretically analyze the limitations of existing memory models within a unified framework and introduce the Stable Hadamard Memory, a novel memory model for reinforcement learning agents. Our model dynamically adjusts memory by erasing no longer needed experiences and reinforcing crucial ones computationally efficiently. To this end, we leverage the Hadamard product for calibrating and updating memory, specifically designed to enhance memory capacity while mitigating numerical and learning challenges. Our approach significantly outperforms state-of-the-art memory-based methods on challenging partially observable benchmarks, such as meta-reinforcement learning, long-horizon credit assignment, and POPGym, demonstrating superior performance in handling long-term and evolving contexts.