WorldEval: World Model as Real-World Robot Policies Evaluator

TL;DR

This paper introduces WorldEval, a scalable and reliable framework using world models to evaluate robot manipulation policies efficiently and safely, correlating well with real-world performance and outperforming existing methods.

Contribution

It presents Policy2Vec for realistic video generation from world models and develops WorldEval, an automated online evaluation pipeline for robot policies.

Findings

Strong correlation between WorldEval and real-world performance

WorldEval outperforms real-to-sim evaluation methods

Enables safe, scalable policy assessment in diverse environments

Abstract

The field of robotics has made significant strides toward developing generalist robot manipulation policies. However, evaluating these policies in real-world scenarios remains time-consuming and challenging, particularly as the number of tasks scales and environmental conditions change. In this work, we demonstrate that world models can serve as a scalable, reproducible, and reliable proxy for real-world robot policy evaluation. A key challenge is generating accurate policy videos from world models that faithfully reflect the robot actions. We observe that directly inputting robot actions or using high-dimensional encoding methods often fails to generate action-following videos. To address this, we propose Policy2Vec, a simple yet effective approach to turn a video generation model into a world simulator that follows latent action to generate the robot video. We then introduce WorldEval, an automated pipeline designed to evaluate real-world robot policies entirely online. WorldEval effectively ranks various robot policies and individual checkpoints within a single policy, and functions as a safety detector to prevent dangerous actions by newly developed robot models. Through comprehensive paired evaluations of manipulation policies in real-world environments, we demonstrate a strong correlation between policy performance in WorldEval and real-world scenarios. Furthermore, our method significantly outperforms popular methods such as real-to-sim approach.