Value from Observations: Towards Large-Scale Imitation Learning via Self-Improvement

TL;DR

This paper advances large-scale imitation learning from observation by developing a method that leverages value functions to learn from nuanced, real-world data distributions, enabling iterative self-improvement.

Contribution

It introduces a novel RL-based imitation learning approach that effectively utilizes action-free demonstrations with complex data distributions for scalable behavior learning.

Findings

The method improves learning from diverse data distributions.

It highlights limitations of existing algorithms in realistic scenarios.

Provides insights for more robust imitation learning techniques.

Abstract

Imitation Learning from Observation (IfO) offers a powerful way to learn behaviors at large-scale: Unlike behavior cloning or offline reinforcement learning, IfO can leverage action-free demonstrations and thus circumvents the need for costly action-labeled demonstrations or reward functions. However, current IfO research focuses on idealized scenarios with mostly bimodal-quality data distributions, restricting the meaningfulness of the results. In contrast, this paper investigates more nuanced distributions and introduces a method to learn from such data, moving closer to a paradigm in which imitation learning can be performed iteratively via self-improvement. Our method adapts RL-based imitation learning to action-free demonstrations, using a value function to transfer information between expert and non-expert data. Through comprehensive evaluation, we delineate the relation between different data distributions and the applicability of algorithms and highlight the limitations of established methods. Our findings provide valuable insights for developing more robust and practical IfO techniques on a path to scalable behaviour learning.