Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learning

TL;DR

This paper introduces a multistep quasimetric learning approach for goal-conditioned reinforcement learning that improves long-horizon planning and stitching in both simulated and real-world robotic tasks using offline data.

Contribution

It presents the first end-to-end offline GCRL method that integrates multistep Monte Carlo returns with quasimetric learning for scalable long-horizon planning.

Findings

Outperforms existing offline GCRL methods on long-horizon tasks with up to 4000 steps

Enables multistep stitching in real-world robotic manipulation from visual observations

Demonstrates robust horizon generalization in complex environments

Abstract

Learning how to reach goals in an environment is a longstanding challenge in AI, yet reasoning over long horizons remains a challenge for modern methods. The key question is how to estimate the temporal distance between pairs of observations. While temporal difference methods leverage local updates to provide optimality guarantees, they often perform worse than Monte Carlo methods that perform global updates (e.g., with multi-step returns), which lack such guarantees. We show how these approaches can be integrated into a practical offline GCRL method that fits a quasimetric distance using a multistep Monte-Carlo return. We show our method outperforms existing offline GCRL methods on long-horizon simulated tasks with up to 4000 steps, even with visual observations. We also demonstrate that our method can enable stitching in the real-world robotic manipulation domain (Bridge setup). Our approach is the first end-to-end offline GCRL method that enables multistep stitching in this real-world manipulation domain from an unlabeled offline dataset of visual observations and demonstrate robust horizon generalization.