CEIL: Generalized Contextual Imitation Learning

TL;DR

CEIL is a versatile imitation learning algorithm that leverages hindsight embeddings to effectively learn from various data sources and settings, demonstrating improved sample efficiency and performance over existing methods.

Contribution

The paper introduces CEIL, a novel generalized imitation learning algorithm that explicitly learns hindsight embeddings and applies across multiple IL settings.

Findings

CEIL outperforms prior methods in sample efficiency on MuJoCo tasks.

CEIL achieves competitive results on offline datasets like D4RL.

CEIL is applicable to diverse IL scenarios including one-shot and cross-domain learning.

Abstract

In this paper, we present \textbf{C}ont\textbf{E}xtual \textbf{I}mitation \textbf{L}earning~(CEIL), a general and broadly applicable algorithm for imitation learning (IL). Inspired by the formulation of hindsight information matching, we derive CEIL by explicitly learning a hindsight embedding function together with a contextual policy using the hindsight embeddings. To achieve the expert matching objective for IL, we advocate for optimizing a contextual variable such that it biases the contextual policy towards mimicking expert behaviors. Beyond the typical learning from demonstrations (LfD) setting, CEIL is a generalist that can be effectively applied to multiple settings including: 1)~learning from observations (LfO), 2)~offline IL, 3)~cross-domain IL (mismatched experts), and 4) one-shot IL settings. Empirically, we evaluate CEIL on the popular MuJoCo tasks (online) and the D4RL dataset (offline). Compared to prior state-of-the-art baselines, we show that CEIL is more sample-efficient in most online IL tasks and achieves better or competitive performances in offline tasks.