Synthetic POMDPs to Challenge Memory-Augmented RL: Memory Demand Structure Modeling

TL;DR

This paper introduces a theoretical framework and methodology for designing customizable, scalable POMDP environments with tunable memory challenges to improve evaluation of memory-augmented RL agents.

Contribution

It presents a new theoretical framework based on Memory Demand Structure and a construction methodology for creating POMDPs with predefined memory challenges.

Findings

Developed a theoretical framework for analyzing POMDPs based on MDS.

Created a methodology to construct POMDPs with specific memory demands.

Provided a suite of scalable POMDP environments with adjustable difficulty.

Abstract

Recent benchmarks for memory-augmented reinforcement learning (RL) have introduced partially observable Markov decision process (POMDP) environments in which agents must use historical observations to make decisions. However, these benchmarks often lack fine-grained control over the challenges posed to memory models. Synthetic environments offer a solution, enabling precise manipulation of environment dynamics for rigorous and interpretable evaluation of memory-augmented RL. This paper advances the design of such customizable POMDPs with three key contributions: (1) a theoretical framework for analyzing POMDPs based on Memory Demand Structure (MDS) and related concepts; (2) a methodology using linear dynamics, state aggregation, and reward redistribution to construct POMDPs with predefined MDS; and (3) a suite of lightweight, scalable POMDP environments with tunable difficulty, grounded in our theoretical insights. Overall, our work clarifies core challenges in partially observable RL, offers principled guidelines for POMDP design, and aids in selecting and developing suitable memory architectures for RL tasks.