ABC: Adversarial Behavioral Cloning for Offline Mode-Seeking Imitation Learning

TL;DR

This paper introduces Adversarial Behavioral Cloning (ABC), a novel method that enhances offline imitation learning by shifting from mean-seeking to mode-seeking behavior using adversarial training, leading to improved policy extraction.

Contribution

The paper proposes ABC, a mode-seeking variant of behavioral cloning that incorporates adversarial training to better capture the expert action distribution.

Findings

ABC outperforms standard behavioral cloning in toy and Hopper domains.

ABC demonstrates mode-seeking behavior, reducing mean-seeking limitations.

Empirical results show improved policy accuracy with ABC.

Abstract

Given a dataset of expert agent interactions with an environment of interest, a viable method to extract an effective agent policy is to estimate the maximum likelihood policy indicated by this data. This approach is commonly referred to as behavioral cloning (BC). In this work, we describe a key disadvantage of BC that arises due to the maximum likelihood objective function; namely that BC is mean-seeking with respect to the state-conditional expert action distribution when the learner's policy is represented with a Gaussian. To address this issue, we introduce a modified version of BC, Adversarial Behavioral Cloning (ABC), that exhibits mode-seeking behavior by incorporating elements of GAN (generative adversarial network) training. We evaluate ABC on toy domains and a domain based on Hopper from the DeepMind Control suite, and show that it outperforms standard BC by being mode-seeking in nature.