Zero-Shot Offline Imitation Learning via Optimal Transport

TL;DR

This paper introduces a novel zero-shot offline imitation learning method that directly optimizes occupancy matching, enabling agents to imitate unseen behaviors from limited data without myopic pitfalls.

Contribution

It proposes a new approach that lifts goal-conditioned value functions to occupancy distances, improving zero-shot imitation from offline data.

Findings

Capable of non-myopic, zero-shot imitation in complex benchmarks

Learns from offline, suboptimal data effectively

Mitigates myopic behavior in imitation learning

Abstract

Zero-shot imitation learning algorithms hold the promise of reproducing unseen behavior from as little as a single demonstration at test time. Existing practical approaches view the expert demonstration as a sequence of goals, enabling imitation with a high-level goal selector, and a low-level goal-conditioned policy. However, this framework can suffer from myopic behavior: the agent's immediate actions towards achieving individual goals may undermine long-term objectives. We introduce a novel method that mitigates this issue by directly optimizing the occupancy matching objective that is intrinsic to imitation learning. We propose to lift a goal-conditioned value function to a distance between occupancies, which are in turn approximated via a learned world model. The resulting method can learn from offline, suboptimal data, and is capable of non-myopic, zero-shot imitation, as we demonstrate in complex, continuous benchmarks. The code is available at https://github.com/martius-lab/zilot.