Laplacian Representations for Decision-Time Planning

TL;DR

This paper introduces Laplacian representations for decision-time planning in model-based RL, enabling effective long-horizon planning and subgoal decomposition, demonstrated by the ALPS hierarchical algorithm outperforming baselines on offline goal-conditioned tasks.

Contribution

The paper proposes Laplacian representations as a novel latent space for planning, and introduces ALPS, a hierarchical planning algorithm leveraging these representations.

Findings

Laplacian representations effectively capture multi-scale state-space distances.

ALPS outperforms baseline methods on offline goal-conditioned RL tasks.

The approach mitigates long-horizon prediction errors.

Abstract

Planning with a learned model remains a key challenge in model-based reinforcement learning (RL). In decision-time planning, state representations are critical as they must support local cost computation while preserving long-horizon structure. In this paper, we show that the Laplacian representation provides an effective latent space for planning by capturing state-space distances at multiple time scales. This representation preserves meaningful distances and naturally decomposes long-horizon problems into subgoals, also mitigating the compounding errors that arise over long prediction horizons. Building on these properties, we introduce ALPS, a hierarchical planning algorithm, and demonstrate that it outperforms commonly used baselines on a selection of offline goal-conditioned RL tasks from OGBench, a benchmark previously dominated by model-free methods.