RoboWM-Bench: A Benchmark for Evaluating World Models in Robotic Manipulation

TL;DR

RoboWM-Bench is a new benchmark that evaluates the physical executability of predicted manipulation videos by translating them into action sequences and testing in simulation environments.

Contribution

It introduces a manipulation-centric benchmark that assesses whether generated videos can be converted into executable actions in physically grounded simulations.

Findings

Visual plausibility does not always imply executability.

Factors like spatial reasoning and contact prediction significantly impact execution success.

The benchmark reveals gaps in current video world models for robotic manipulation.

Abstract

Recent advances in large-scale video world models have enabled increasingly realistic future prediction, raising the prospect of using generated videos as scalable supervision for robot learning. However, for embodied manipulation, perceptual realism alone is not sufficient: generated interactions must also be physically consistent and executable by robotic agents. Existing benchmarks provide valuable assessments of visual quality and physical plausibility, but they do not systematically evaluate whether predicted behaviors can be translated into executable actions that complete manipulation tasks. We introduce RoboWM-Bench, a manipulation-centric benchmark for embodiment-grounded evaluation of video world models. RoboWM-Bench converts generated human-hand and robotic manipulation videos into embodied action sequences and validates them through execution in physically grounded simulation environments. Built on real-to-sim scene reconstruction and diverse manipulation tasks, RoboWM-Bench enables standardized, reproducible, and scalable evaluation of physical executability. Using RoboWM-Bench, we evaluate state-of-the-art video world models and observe that visual plausibility and embodied executability are not always aligned. Our analysis highlights several recurring factors that affect execution performance, including spatial reasoning, contact prediction, and non-physical geometric distortions, particularly in complex and long-horizon interactions. These findings provide a more fine-grained view of current model capabilities and underscore the value of embodiment-aware evaluation for guiding physically grounded world modeling in robotic manipulation.