Imitation with Neural Density Models

TL;DR

This paper introduces Neural Density Imitation, a novel imitation learning framework that estimates expert behavior density and uses it as a reward in reinforcement learning, achieving high efficiency on control benchmarks.

Contribution

It presents a new density-based imitation learning method that is non-adversarial, model-free, and provably bounds divergence between expert and imitator.

Findings

Achieves state-of-the-art demonstration efficiency

Provides a practical algorithm for density-based imitation learning

Proven theoretical bounds on divergence minimization

Abstract

We propose a new framework for Imitation Learning (IL) via density estimation of the expert's occupancy measure followed by Maximum Occupancy Entropy Reinforcement Learning (RL) using the density as a reward. Our approach maximizes a non-adversarial model-free RL objective that provably lower bounds reverse Kullback-Leibler divergence between occupancy measures of the expert and imitator. We present a practical IL algorithm, Neural Density Imitation (NDI), which obtains state-of-the-art demonstration efficiency on benchmark control tasks.