Wow, wo, val! A Comprehensive Embodied World Model Evaluation Turing Test

TL;DR

This paper introduces the WoW-wo-val benchmark to evaluate the perceptual fidelity, robustness, and generalization of video foundation models for embodied AI, revealing significant gaps between generated videos and real-world performance.

Contribution

It establishes a standardized evaluation framework with 22 metrics for assessing embodied world models, including a human-aligned Turing Test and real-world execution benchmarks.

Findings

Models show limited spatiotemporal consistency and physical reasoning.

High correlation between model scores and human preferences (>0.93).

Most models fail in real-world execution, with success rates around 0%, while WoW achieves 40.74%.

Abstract

As world models gain momentum in Embodied AI, an increasing number of works explore using video foundation models as predictive world models for downstream embodied tasks like 3D prediction or interactive generation. However, before exploring these downstream tasks, video foundation models still have two critical questions unanswered: (1) whether their generative generalization is sufficient to maintain perceptual fidelity in the eyes of human observers, and (2) whether they are robust enough to serve as a universal prior for real-world embodied agents. To provide a standardized framework for answering these questions, we introduce the Embodied Turing Test benchmark: WoW-World-Eval (Wow,wo,val). Building upon 609 robot manipulation data, Wow-wo-val examines five core abilities, including perception, planning, prediction, generalization, and execution. We propose a comprehensive evaluation protocol with 22 metrics to assess the models' generation ability, which achieves a high Pearson Correlation between the overall score and human preference (>0.93) and establishes a reliable foundation for the Human Turing Test. On Wow-wo-val, models achieve only 17.27 on long-horizon planning and at best 68.02 on physical consistency, indicating limited spatiotemporal consistency and physical reasoning. For the Inverse Dynamic Model Turing Test, we first use an IDM to evaluate the video foundation models' execution accuracy in the real world. However, most models collapse to $\approx$ 0% success, while WoW maintains a 40.74% success rate. These findings point to a noticeable gap between the generated videos and the real world, highlighting the urgency and necessity of benchmarking World Model in Embodied AI.