Memory Allocation in Resource-Constrained Reinforcement Learning

TL;DR

This paper investigates how memory limitations affect reinforcement learning agents' performance, analyzing the trade-offs in memory allocation between modeling and planning in resource-constrained environments.

Contribution

It introduces an analysis of memory allocation strategies in resource-constrained reinforcement learning, focusing on MCTS and DQN algorithms.

Findings

Memory allocation impacts agent performance significantly.

Different environments require different memory strategies.

Trade-offs between modeling and planning are crucial for efficiency.

Abstract

Resource constraints can fundamentally change both learning and decision-making. We explore how memory constraints influence an agent's performance when navigating unknown environments using standard reinforcement learning algorithms. Specifically, memory-constrained agents face a dilemma: how much of their limited memory should be allocated to each of the agent's internal processes, such as estimating a world model, as opposed to forming a plan using that model? We study this dilemma in MCTS- and DQN-based algorithms and examine how different allocations of memory impact performance in episodic and continual learning settings.