A Coupled Flow Approach to Imitation Learning

TL;DR

This paper introduces CFIL, a novel imitation learning method that explicitly models state distributions using coupled normalizing flows, achieving state-of-the-art results with minimal expert data.

Contribution

It proposes a new coupled flow model for explicit state distribution estimation in imitation learning, extending to various data regimes.

Findings

Achieves state-of-the-art performance on benchmark tasks.

Effective with only a single expert trajectory.

Extends to subsampled and state-only settings.

Abstract

In reinforcement learning and imitation learning, an object of central importance is the state distribution induced by the policy. It plays a crucial role in the policy gradient theorem, and references to it--along with the related state-action distribution--can be found all across the literature. Despite its importance, the state distribution is mostly discussed indirectly and theoretically, rather than being modeled explicitly. The reason being an absence of appropriate density estimation tools. In this work, we investigate applications of a normalizing flow-based model for the aforementioned distributions. In particular, we use a pair of flows coupled through the optimality point of the Donsker-Varadhan representation of the Kullback-Leibler (KL) divergence, for distribution matching based imitation learning. Our algorithm, Coupled Flow Imitation Learning (CFIL), achieves state-of-the-art performance on benchmark tasks with a single expert trajectory and extends naturally to a variety of other settings, including the subsampled and state-only regimes.