Horizon Generalization in Reinforcement Learning

TL;DR

This paper explores horizon generalization in goal-conditioned reinforcement learning, proposing that policies trained on nearby goals can generalize to distant goals through planning invariance, supported by theoretical analysis and experimental evidence.

Contribution

It introduces the concept of horizon generalization in RL, linking it to planning invariance, and provides theoretical proof and experimental support for this property.

Findings

Horizon generalization is theoretically achievable under certain assumptions.

Policies trained on nearby goals can generalize to distant goals via planning invariance.

Experimental results support the theoretical claims about horizon generalization.

Abstract

We study goal-conditioned RL through the lens of generalization, but not in the traditional sense of random augmentations and domain randomization. Rather, we aim to learn goal-directed policies that generalize with respect to the horizon: after training to reach nearby goals (which are easy to learn), these policies should succeed in reaching distant goals (which are quite challenging to learn). In the same way that invariance is closely linked with generalization is other areas of machine learning (e.g., normalization layers make a network invariant to scale, and therefore generalize to inputs of varying scales), we show that this notion of horizon generalization is closely linked with invariance to planning: a policy navigating towards a goal will select the same actions as if it were navigating to a waypoint en route to that goal. Thus, such a policy trained to reach nearby goals should succeed at reaching arbitrarily-distant goals. Our theoretical analysis proves that both horizon generalization and planning invariance are possible, under some assumptions. We present new experimental results and recall findings from prior work in support of our theoretical results. Taken together, our results open the door to studying how techniques for invariance and generalization developed in other areas of machine learning might be adapted to achieve this alluring property.