Multi-scale Predictive Representations for Goal-conditioned Reinforcement Learning

TL;DR

This paper introduces Ms.PR, a multi-scale predictive framework that enhances goal-conditioned reinforcement learning by aligning state and goal representations across various scales, improving robustness and performance.

Contribution

The paper proposes a novel multi-scale predictive supervision method, Ms.PR, to improve representation learning and stability in offline goal-conditioned reinforcement learning.

Findings

Ms.PR improves representation quality in GCRL tasks.

Ms.PR achieves state-of-the-art performance across diverse tasks.

Ms.PR remains robust under noisy and challenging data regimes.

Abstract

This paper investigates robust representation learning in offline goal-conditioned reinforcement learning (GCRL). Particularly in sparse reward scenarios, learning representations that align state and goal latents is a challenge that frequently culminates in representation divergence where the encoder drifts toward a low-dimensional, goal-agnostic subspace that destabilizes policy learning. We address this issue by showing that an agent must acquire a fundamental understanding of its environment across multiple scales, from local physical dynamics to long-horizon goal-directed structure. Building on this insight, we propose Ms.PR, a framework that leverages multi-scale predictive supervision to enforce goal-directed alignment within the latent space. We demonstrate that Ms.PR leads to improved representation quality and strong performance on both vision and state-based tasks. Furthermore, we show that our approach is exceptionally resilient under realistic, challenging data regimes, maintaining state-of-the-art performance across a wide variety of tasks, trajectory stitching scenarios, and extreme noise conditions.