Robust Imitation of Diverse Behaviors

TL;DR

This paper introduces a novel approach combining variational autoencoders and GAIL to achieve robust, diverse imitation learning for complex motor control tasks, overcoming mode collapse and improving performance with limited demonstrations.

Contribution

The authors develop a new method that integrates semantic policy embeddings with GAIL, enhancing robustness and diversity in imitation learning for high-dimensional robotic tasks.

Findings

Learned semantic policy embeddings on a 9 DoF robot arm.

Achieved smooth interpolation of behaviors via embeddings.

Demonstrated improved robustness and diversity in gait imitation.

Abstract

Deep generative models have recently shown great promise in imitation learning for motor control. Given enough data, even supervised approaches can do one-shot imitation learning; however, they are vulnerable to cascading failures when the agent trajectory diverges from the demonstrations. Compared to purely supervised methods, Generative Adversarial Imitation Learning (GAIL) can learn more robust controllers from fewer demonstrations, but is inherently mode-seeking and more difficult to train. In this paper, we show how to combine the favourable aspects of these two approaches. The base of our model is a new type of variational autoencoder on demonstration trajectories that learns semantic policy embeddings. We show that these embeddings can be learned on a 9 DoF Jaco robot arm in reaching tasks, and then smoothly interpolated with a resulting smooth interpolation of reaching behavior. Leveraging these policy representations, we develop a new version of GAIL that (1) is much more robust than the purely-supervised controller, especially with few demonstrations, and (2) avoids mode collapse, capturing many diverse behaviors when GAIL on its own does not. We demonstrate our approach on learning diverse gaits from demonstration on a 2D biped and a 62 DoF 3D humanoid in the MuJoCo physics environment.