Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learning

TL;DR

This paper introduces goal-conditioned bisimulation, a novel state abstraction method that enables goal generalization in reinforcement learning by inferring goals from analogous tasks, demonstrated through simulation manipulation tasks.

Contribution

It proposes a new form of state abstraction called goal-conditioned bisimulation, and develops a metric learning approach to generalize to new goals in RL tasks.

Findings

Successfully generalizes to new goals in simulation tasks

Representation is sufficient for downstream state-only reward tasks

Achieves skill reuse through functional equivariance

Abstract

Building generalizable goal-conditioned agents from rich observations is a key to reinforcement learning (RL) solving real world problems. Traditionally in goal-conditioned RL, an agent is provided with the exact goal they intend to reach. However, it is often not realistic to know the configuration of the goal before performing a task. A more scalable framework would allow us to provide the agent with an example of an analogous task, and have the agent then infer what the goal should be for its current state. We propose a new form of state abstraction called goal-conditioned bisimulation that captures functional equivariance, allowing for the reuse of skills to achieve new goals. We learn this representation using a metric form of this abstraction, and show its ability to generalize to new goals in simulation manipulation tasks. Further, we prove that this learned representation is sufficient not only for goal conditioned tasks, but is amenable to any downstream task described by a state-only reward function. Videos can be found at https://sites.google.com/view/gc-bisimulation.