Scalable Policy Evaluation with Video World Models

TL;DR

This paper proposes using action-conditional video world models trained on internet videos to evaluate robotic policies efficiently, reducing reliance on costly real-world testing and simulation environments.

Contribution

It introduces a scalable, video-based world model approach for policy evaluation that leverages pre-trained models and internet videos, addressing data collection and sim-to-real gap issues.

Findings

Models correlate well with actual policy values

Effective across multiple evaluation metrics

Reduces need for real-world robot testing

Abstract

Training generalist policies for robotic manipulation has shown great promise, as they enable language-conditioned, multi-task behaviors across diverse scenarios. However, evaluating these policies remains difficult because real-world testing is expensive, time-consuming, and labor-intensive. It also requires frequent environment resets and carries safety risks when deploying unproven policies on physical robots. Manually creating and populating simulation environments with assets for robotic manipulation has not addressed these issues, primarily due to the significant engineering effort required and the substantial sim-to-real gap, both in terms of physics and rendering. In this paper, we explore the use of action-conditional video generation models as a scalable way to learn world models for policy evaluation. We demonstrate how to incorporate action conditioning into existing pre-trained video generation models. This allows leveraging internet-scale in-the-wild online videos during the pre-training stage and alleviates the need for a large dataset of paired video-action data, which is expensive to collect for robotic manipulation. Our paper examines the effect of dataset diversity, pre-trained weights, and common failure cases for the proposed evaluation pipeline. Our experiments demonstrate that across various metrics, including policy ranking and the correlation between actual policy values and predicted policy values, these models offer a promising approach for evaluating policies without requiring real-world interactions.