Towards High-Consistency Embodied World Model with Multi-View Trajectory Videos

TL;DR

This paper introduces MTV-World, a multi-view embodied world model that enhances visuomotor prediction and control accuracy by leveraging multi-view trajectory videos and an auto-evaluation pipeline, addressing spatial information loss issues.

Contribution

It proposes a novel multi-view framework for embodied world modeling that improves spatial consistency and control precision in complex robotic scenarios.

Findings

MTV-World achieves high control accuracy in dual-arm robotic tasks.

The multi-view approach compensates for spatial information loss in trajectory control.

The auto-evaluation pipeline effectively measures control and interaction accuracy.

Abstract

Embodied world models aim to predict and interact with the physical world through visual observations and actions. However, existing models struggle to accurately translate low-level actions (e.g., joint positions) into precise robotic movements in predicted frames, leading to inconsistencies with real-world physical interactions. To address these limitations, we propose MTV-World, an embodied world model that introduces Multi-view Trajectory-Video control for precise visuomotor prediction. Specifically, instead of directly using low-level actions for control, we employ trajectory videos obtained through camera intrinsic and extrinsic parameters and Cartesian-space transformation as control signals. However, projecting 3D raw actions onto 2D images inevitably causes a loss of spatial information, making a single view insufficient for accurate interaction modeling. To overcome this, we introduce a multi-view framework that compensates for spatial information loss and ensures high-consistency with physical world. MTV-World forecasts future frames based on multi-view trajectory videos as input and conditioning on an initial frame per view. Furthermore, to systematically evaluate both robotic motion precision and object interaction accuracy, we develop an auto-evaluation pipeline leveraging multimodal large models and referring video object segmentation models. To measure spatial consistency, we formulate it as an object location matching problem and adopt the Jaccard Index as the evaluation metric. Extensive experiments demonstrate that MTV-World achieves precise control execution and accurate physical interaction modeling in complex dual-arm scenarios.