Reinforcement Learning under Latent Dynamics: Toward Statistical and Algorithmic Modularity

TL;DR

This paper explores the statistical and algorithmic challenges of reinforcement learning in environments with complex observations but simple underlying dynamics, proposing conditions for tractability and reductions for practical algorithms.

Contribution

It provides the first unified statistical and algorithmic framework for RL under general latent dynamics, including negative and positive results and new reduction techniques.

Findings

Most RL with rich observations is statistically intractable without specific conditions.

Latent pushforward coverability enables statistical tractability in complex environments.

Develops efficient observable-to-latent reductions for RL algorithms.

Abstract

Real-world applications of reinforcement learning often involve environments where agents operate on complex, high-dimensional observations, but the underlying (''latent'') dynamics are comparatively simple. However, outside of restrictive settings such as small latent spaces, the fundamental statistical requirements and algorithmic principles for reinforcement learning under latent dynamics are poorly understood. This paper addresses the question of reinforcement learning under $\textit{general}$ latent dynamics from a statistical and algorithmic perspective. On the statistical side, our main negative result shows that most well-studied settings for reinforcement learning with function approximation become intractable when composed with rich observations; we complement this with a positive result, identifying latent pushforward coverability as a general condition that enables statistical tractability. Algorithmically, we develop provably efficient observable-to-latent reductions -- that is, reductions that transform an arbitrary algorithm for the latent MDP into an algorithm that can operate on rich observations -- in two settings: one where the agent has access to hindsight observations of the latent dynamics [LADZ23], and one where the agent can estimate self-predictive latent models [SAGHCB20]. Together, our results serve as a first step toward a unified statistical and algorithmic theory for reinforcement learning under latent dynamics.