Latent Policies for Adversarial Imitation Learning

TL;DR

This paper introduces a stable adversarial imitation learning method that uses a latent action space to improve training stability and performance in high-dimensional robot tasks.

Contribution

It proposes LAPAL, a novel approach that employs a latent action space via an encoder-decoder model to enhance GAIL training stability and effectiveness.

Findings

LAPAL training is stable with near-monotonic performance improvements.

LAPAL achieves expert-level performance in complex locomotion and manipulation tasks.

Compared to GAIL, LAPAL converges faster and performs better in high-dimensional environments.

Abstract

This paper considers learning robot locomotion and manipulation tasks from expert demonstrations. Generative adversarial imitation learning (GAIL) trains a discriminator that distinguishes expert from agent transitions, and in turn use a reward defined by the discriminator output to optimize a policy generator for the agent. This generative adversarial training approach is very powerful but depends on a delicate balance between the discriminator and the generator training. In high-dimensional problems, the discriminator training may easily overfit or exploit associations with task-irrelevant features for transition classification. A key insight of this work is that performing imitation learning in a suitable latent task space makes the training process stable, even in challenging high-dimensional problems. We use an action encoder-decoder model to obtain a low-dimensional latent action space and train a LAtent Policy using Adversarial imitation Learning (LAPAL). The encoder-decoder model can be trained offline from state-action pairs to obtain a task-agnostic latent action representation or online, simultaneously with the discriminator and generator training, to obtain a task-aware latent action representation. We demonstrate that LAPAL training is stable, with near-monotonic performance improvement, and achieves expert performance in most locomotion and manipulation tasks, while a GAIL baseline converges slower and does not achieve expert performance in high-dimensional environments.