When Is Generalizable Reinforcement Learning Tractable?

TL;DR

This paper investigates the conditions under which reinforcement learning agents can efficiently generalize across multiple environments, introducing structural conditions that delineate when such generalization is tractable or impossible.

Contribution

The paper defines Weak Proximity and Strong Proximity conditions, proving the impossibility of tractable generalization under the former and sufficiency under the latter.

Findings

Weak Proximity does not guarantee tractable generalization.

Strong Proximity ensures efficient generalization.

Even with shared structure, worst-case scenarios prevent generalization.

Abstract

Agents trained by reinforcement learning (RL) often fail to generalize beyond the environment they were trained in, even when presented with new scenarios that seem similar to the training environment. We study the query complexity required to train RL agents that generalize to multiple environments. Intuitively, tractable generalization is only possible when the environments are similar or close in some sense. To capture this, we introduce Weak Proximity, a natural structural condition that requires the environments to have highly similar transition and reward functions and share a policy providing optimal value. Despite such shared structure, we prove that tractable generalization is impossible in the worst case. This holds even when each individual environment can be efficiently solved to obtain an optimal linear policy, and when the agent possesses a generative model. Our lower bound applies to the more complex task of representation learning for the purpose of efficient generalization to multiple environments. On the positive side, we introduce Strong Proximity, a strengthened condition which we prove is sufficient for efficient generalization.