Imitation by Predicting Observations

TL;DR

This paper introduces FORM, a novel imitation learning method that learns from observations alone, matching expert performance and demonstrating robustness to irrelevant observations in continuous control tasks.

Contribution

The paper proposes FORM, a new observation-only imitation learning approach derived from inverse RL, which does not require ground truth actions and is robust to extraneous observations.

Findings

FORM matches GAIL's performance on DeepMind Control Suite

FORM outperforms GAIL with task-irrelevant features

The method is effective for continuous control tasks

Abstract

Imitation learning enables agents to reuse and adapt the hard-won expertise of others, offering a solution to several key challenges in learning behavior. Although it is easy to observe behavior in the real-world, the underlying actions may not be accessible. We present a new method for imitation solely from observations that achieves comparable performance to experts on challenging continuous control tasks while also exhibiting robustness in the presence of observations unrelated to the task. Our method, which we call FORM (for "Future Observation Reward Model") is derived from an inverse RL objective and imitates using a model of expert behavior learned by generative modelling of the expert's observations, without needing ground truth actions. We show that FORM performs comparably to a strong baseline IRL method (GAIL) on the DeepMind Control Suite benchmark, while outperforming GAIL in the presence of task-irrelevant features.