Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representations

TL;DR

This paper introduces BCV-LR, a novel unsupervised framework that enables highly sample-efficient imitation learning from videos by extracting and aligning latent action representations, outperforming existing methods in visual control tasks.

Contribution

The paper presents the first method to achieve sample-efficient visual policy learning directly from videos without any additional supervision, using latent representations and iterative policy refinement.

Findings

Outperforms state-of-the-art ILV and RL methods in sample efficiency.

Enables expert-level performance on some visual control tasks.

Demonstrates that videos alone can support highly efficient policy learning.

Abstract

Humans can efficiently extract knowledge and learn skills from the videos within only a few trials and errors. However, it poses a big challenge to replicate this learning process for autonomous agents, due to the complexity of visual input, the absence of action or reward signals, and the limitations of interaction steps. In this paper, we propose a novel, unsupervised, and sample-efficient framework to achieve imitation learning from videos (ILV), named Behavior Cloning from Videos via Latent Representations (BCV-LR). BCV-LR extracts action-related latent features from high-dimensional video inputs through self-supervised tasks, and then leverages a dynamics-based unsupervised objective to predict latent actions between consecutive frames. The pre-trained latent actions are fine-tuned and efficiently aligned to the real action space online (with collected interactions) for policy behavior cloning. The cloned policy in turn enriches the agent experience for further latent action finetuning, resulting in an iterative policy improvement that is highly sample-efficient. We conduct extensive experiments on a set of challenging visual tasks, including both discrete control and continuous control. BCV-LR enables effective (even expert-level on some tasks) policy performance with only a few interactions, surpassing state-of-the-art ILV baselines and reinforcement learning methods (provided with environmental rewards) in terms of sample efficiency across 24/28 tasks. To the best of our knowledge, this work for the first time demonstrates that videos can support extremely sample-efficient visual policy learning, without the need to access any other expert supervision.