Compositional Transduction with Latent Analogies for Offline Goal-Conditioned Reinforcement Learning

TL;DR

This paper introduces a novel analogy-based approach for offline goal-conditioned reinforcement learning, enabling compositional generalization to unseen goal-context pairs and outperforming prior methods.

Contribution

It formalizes analogy transduction for compositional goal-reaching and proposes a new representation and method to generalize beyond seen context pairs in offline GCRL.

Findings

Significantly outperforms prior methods on OGBench manipulation environments.

Introduces a new analogy representation capturing invariant changes under optimal execution.

Enables generalization to unseen analogy-context pairs in offline GCRL.

Abstract

Compositional generalization is essential for reaching unseen goals under novel contextual variations in offline goal-conditioned reinforcement learning (GCRL), where a generalist goal-reaching agent must be learned from limited data. Most prior approaches pursue this via trajectory stitching over temporally contiguous segments, which limits composing behaviors across varying contexts. To overcome this limitation, we formalize analogy transduction as synthesizing new plans by composing task-endogenous analogies with given contexts and propose a novel analogy representation tailored for it. Grounded in our theory, this analogy representation captures what changes under optimal task execution, remains invariant to contextual variations, and is sufficient for optimal goal reaching. We further contend that generalization to unseen analogy-context pairs is a practical obstacle in analogy transduction, and introduce a new approach for offline GCRL that enables analogy transduction beyond seen pairs to unseen combinations. We empirically demonstrate the effectiveness of our approach on OGBench manipulation environments, substantially outperforming prior methods that do not perform analogy transduction. Project page: https://rllab-snu.github.io/projects/CTA/