ACWM-Phys: Investigating Generalized Physical Interaction in Action-Conditioned Video World Models

TL;DR

ACWM-Phys introduces a comprehensive benchmark for evaluating action-conditioned video world models across diverse physical dynamics, highlighting challenges in out-of-distribution generalization and guiding future model improvements.

Contribution

The paper presents ACWM-Phys, a new benchmark environment with diverse physical interactions and evaluation protocols for assessing generalized physical prediction in world models.

Findings

Models perform well on simple, geometric interactions.

Generalization drops significantly with deformable and complex motions.

Cross-attention and causal VAEs improve modeling high-dimensional actions.

Abstract

Action-conditioned world models (ACWMs) have shown strong promise for video prediction and decision-making. However, existing benchmarks are largely restricted to egocentric navigation or narrow, task-specific robotics datasets, offering only limited coverage of the rich physical interactions required for generalized world understanding. We introduce ACWM-Phys, a new benchmark for evaluating action-conditioned prediction under diverse physical dynamics in a clean, controllable simulation environment with a carefully designed action space. ACWM-Phys contains training and evaluation data spanning rigid-body dynamics, kinematics, deformable-object interactions, and particle dynamics. To evaluate both interpolation and generalization, we design in-distribution and out-of-distribution protocols with controlled shifts in interaction patterns or scene configurations. By building the benchmark in a fully controllable simulator, ACWM-Phys enables precise data collection, reproducible evaluation, and systematic analysis of model capabilities for physically grounded world modeling. Through systematic experiments on ACWM-DiT, we find that OoD generalization depends not only on the physical regime but also on effective task complexity: models generalize well on visually simple, low-dimensional interactions with clear geometric structure, but suffer larger drops on deformable contacts, high-dimensional control, and complex articulated motion. This suggests that the model still relies heavily on visual appearance patterns instead of fully learning the underlying physics. Ablations show that cross-attention improves high-dimensional action conditioning, causal VAEs outperform frame-wise encoders, and larger action spaces are harder to model but can improve generalization by providing richer control signals. These findings guide the design of physically grounded world models.