Goal-Conditioned Predictive Coding for Offline Reinforcement Learning

TL;DR

This paper introduces Goal-Conditioned Predictive Coding (GCPC), a sequence modeling approach that encodes trajectories into useful representations, significantly improving offline reinforcement learning performance across various challenging environments.

Contribution

The paper proposes GCPC, a novel sequence modeling objective that enhances trajectory representations for goal-conditioned policies in offline RL, unifying and advancing existing methods.

Findings

GCPC produces powerful trajectory representations.

Sequence modeling improves decision-making in complex tasks.

GCPC achieves competitive results on multiple benchmarks.

Abstract

Recent work has demonstrated the effectiveness of formulating decision making as supervised learning on offline-collected trajectories. Powerful sequence models, such as GPT or BERT, are often employed to encode the trajectories. However, the benefits of performing sequence modeling on trajectory data remain unclear. In this work, we investigate whether sequence modeling has the ability to condense trajectories into useful representations that enhance policy learning. We adopt a two-stage framework that first leverages sequence models to encode trajectory-level representations, and then learns a goal-conditioned policy employing the encoded representations as its input. This formulation allows us to consider many existing supervised offline RL methods as specific instances of our framework. Within this framework, we introduce Goal-Conditioned Predictive Coding (GCPC), a sequence modeling objective that yields powerful trajectory representations and leads to performant policies. Through extensive empirical evaluations on AntMaze, FrankaKitchen and Locomotion environments, we observe that sequence modeling can have a significant impact on challenging decision making tasks. Furthermore, we demonstrate that GCPC learns a goal-conditioned latent representation encoding the future trajectory, which enables competitive performance on all three benchmarks.