What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?

TL;DR

This paper investigates the factors influencing the ability of offline goal-conditioned reinforcement learning to generalize to unseen goals, providing theoretical insights and proposing a new method that outperforms existing approaches on a comprehensive benchmark.

Contribution

The paper offers the first thorough study of out-of-distribution generalization in offline GCRL, deriving a theory and proposing the GOAT algorithm that improves unseen goal achievement.

Findings

Weighted imitation learning shows better generalization than pessimism-based methods.

GOAT outperforms state-of-the-art methods on a new benchmark with 9 IID and 17 OOD tasks.

Theoretical analysis highlights key design choices for OOD generalization.

Abstract

Offline goal-conditioned RL (GCRL) offers a way to train general-purpose agents from fully offline datasets. In addition to being conservative within the dataset, the generalization ability to achieve unseen goals is another fundamental challenge for offline GCRL. However, to the best of our knowledge, this problem has not been well studied yet. In this paper, we study out-of-distribution (OOD) generalization of offline GCRL both theoretically and empirically to identify factors that are important. In a number of experiments, we observe that weighted imitation learning enjoys better generalization than pessimism-based offline RL method. Based on this insight, we derive a theory for OOD generalization, which characterizes several important design choices. We then propose a new offline GCRL method, Generalizable Offline goAl-condiTioned RL (GOAT), by combining the findings from our theoretical and empirical studies. On a new benchmark containing 9 independent identically distributed (IID) tasks and 17 OOD tasks, GOAT outperforms current state-of-the-art methods by a large margin.