Memory Lens: How Much Memory Does an Agent Use?

TL;DR

This paper introduces a method to quantify the memory usage of reinforcement learning agents by estimating mutual information between past behaviors and current actions, providing insights into their internal memory mechanisms.

Contribution

It presents a novel, implementation-independent approach to measure the minimal memory capacity of RL policies through mutual information estimation, validated on Atari games.

Findings

Memory usage varies significantly across different Atari games.

The method provides a lower bound on the minimal memory capacity of policies.

Memory estimation reveals insights into agent behavior and policy complexity.

Abstract

We propose a new method to study the internal memory used by reinforcement learning policies. We estimate the amount of relevant past information by estimating mutual information between behavior histories and the current action of an agent. We perform this estimation in the passive setting, that is, we do not intervene but merely observe the natural behavior of the agent. Moreover, we provide a theoretical justification for our approach by showing that it yields an implementation-independent lower bound on the minimal memory capacity of any agent that implement the observed policy. We demonstrate our approach by estimating the use of memory of DQN policies on concatenated Atari frames, demonstrating sharply different use of memory across 49 games. The study of memory as information that flows from the past to the current action opens avenues to understand and improve successful reinforcement learning algorithms.