Model-Free Imitation Learning with Policy Optimization

TL;DR

This paper introduces a model-free imitation learning algorithm using policy gradients that efficiently learns policies in high-dimensional environments by mimicking expert demonstrations without solving complex planning problems.

Contribution

It presents a novel model-free, policy-gradient-based imitation learning method that guarantees convergence and scales to large, continuous environments.

Findings

Scales to high-dimensional, continuous environments

Guarantees convergence to local minima

Performs at least as well as expert policies

Abstract

In imitation learning, an agent learns how to behave in an environment with an unknown cost function by mimicking expert demonstrations. Existing imitation learning algorithms typically involve solving a sequence of planning or reinforcement learning problems. Such algorithms are therefore not directly applicable to large, high-dimensional environments, and their performance can significantly degrade if the planning problems are not solved to optimality. Under the apprenticeship learning formalism, we develop alternative model-free algorithms for finding a parameterized stochastic policy that performs at least as well as an expert policy on an unknown cost function, based on sample trajectories from the expert. Our approach, based on policy gradients, scales to large continuous environments with guaranteed convergence to local minima.