Imitation from Diverse Behaviors: Wasserstein Quality Diversity Imitation Learning with Single-Step Archive Exploration

TL;DR

This paper introduces Wasserstein Quality Diversity Imitation Learning (WQDIL), a novel approach that enhances diversity and stability in imitation learning by combining Wasserstein Auto-Encoder techniques with behavior exploration, outperforming existing methods.

Contribution

WQDIL is the first method to integrate Wasserstein Auto-Encoder based adversarial training with quality diversity imitation learning, addressing stability and overfitting issues.

Findings

Achieves near-expert or beyond-expert quality diversity performance on MuJoCo tasks.

Significantly outperforms state-of-the-art imitation learning methods.

Effectively mitigates behavior-overfitting with a measure-conditioned reward function.

Abstract

Learning diverse and high-performance behaviors from a limited set of demonstrations is a grand challenge. Traditional imitation learning methods usually fail in this task because most of them are designed to learn one specific behavior even with multiple demonstrations. Therefore, novel techniques for \textit{quality diversity imitation learning}, which bridges the quality diversity optimization and imitation learning methods, are needed to solve the above challenge. This work introduces Wasserstein Quality Diversity Imitation Learning (WQDIL), which 1) improves the stability of imitation learning in the quality diversity setting with latent adversarial training based on a Wasserstein Auto-Encoder (WAE), and 2) mitigates a behavior-overfitting issue using a measure-conditioned reward function with a single-step archive exploration bonus. Empirically, our method significantly outperforms state-of-the-art IL methods, achieving near-expert or beyond-expert QD performance on the challenging continuous control tasks derived from MuJoCo environments.