Imitation Learning by State-Only Distribution Matching

TL;DR

This paper introduces a non-adversarial imitation learning method that matches state transition distributions, improving robustness and sample efficiency, with a reliable convergence metric, achieving state-of-the-art results in continuous control tasks.

Contribution

It proposes a novel non-adversarial approach for imitation learning from observations using KL divergence minimization and density models, enhancing stability and interpretability.

Findings

Achieves state-of-the-art performance on continuous control benchmarks.

Provides a reliable convergence and performance estimation metric.

Demonstrates improved robustness and sample efficiency over adversarial methods.

Abstract

Imitation Learning from observation describes policy learning in a similar way to human learning. An agent's policy is trained by observing an expert performing a task. While many state-only imitation learning approaches are based on adversarial imitation learning, one main drawback is that adversarial training is often unstable and lacks a reliable convergence estimator. If the true environment reward is unknown and cannot be used to select the best-performing model, this can result in bad real-world policy performance. We propose a non-adversarial learning-from-observations approach, together with an interpretable convergence and performance metric. Our training objective minimizes the Kulback-Leibler divergence (KLD) between the policy and expert state transition trajectories which can be optimized in a non-adversarial fashion. Such methods demonstrate improved robustness when learned density models guide the optimization. We further improve the sample efficiency by rewriting the KLD minimization as the Soft Actor Critic objective based on a modified reward using additional density models that estimate the environment's forward and backward dynamics. Finally, we evaluate the effectiveness of our approach on well-known continuous control environments and show state-of-the-art performance while having a reliable performance estimator compared to several recent learning-from-observation methods.