Sample-efficient Unsupervised Policy Cloning from Ensemble Self-supervised Labeled Videos

TL;DR

This paper introduces UPESV, a novel framework enabling machines to learn policies from unlabeled videos without rewards or expert supervision, mimicking human learning from internet videos efficiently.

Contribution

UPESV is the first method to learn policies from action-free videos using self-supervised tasks for robust dynamics understanding and policy cloning without rewards.

Findings

Achieves state-of-the-art performance in 12 out of 16 environments.

Outperforms five baseline methods on interaction-limited policy learning.

Learns effective policies solely from unlabeled videos without additional supervision.

Abstract

Current advanced policy learning methodologies have demonstrated the ability to develop expert-level strategies when provided enough information. However, their requirements, including task-specific rewards, action-labeled expert trajectories, and huge environmental interactions, can be expensive or even unavailable in many scenarios. In contrast, humans can efficiently acquire skills within a few trials and errors by imitating easily accessible internet videos, in the absence of any other supervision. In this paper, we try to let machines replicate this efficient watching-and-learning process through Unsupervised Policy from Ensemble Self-supervised labeled Videos (UPESV), a novel framework to efficiently learn policies from action-free videos without rewards and any other expert supervision. UPESV trains a video labeling model to infer the expert actions in expert videos through several organically combined self-supervised tasks. Each task performs its duties, and they together enable the model to make full use of both action-free videos and reward-free interactions for robust dynamics understanding and advanced action prediction. Simultaneously, UPESV clones a policy from the labeled expert videos, in turn collecting environmental interactions for self-supervised tasks. After a sample-efficient, unsupervised, and iterative training process, UPESV obtains an advanced policy based on a robust video labeling model. Extensive experiments in sixteen challenging procedurally generated environments demonstrate that the proposed UPESV achieves state-of-the-art interaction-limited policy learning performance (outperforming five current advanced baselines on 12/16 tasks) without exposure to any other supervision except for videos.