Phys4D: Fine-Grained Physics-Consistent 4D Modeling from Video Diffusion

TL;DR

Phys4D introduces a three-stage training pipeline that enhances video diffusion models with physics-consistent 4D representations, improving physical plausibility and dynamic coherence in generated videos.

Contribution

The paper proposes Phys4D, a novel method combining large-scale pretraining, physics-grounded fine-tuning, and reinforcement learning to achieve fine-grained physical consistency in 4D video modeling.

Findings

Significant improvement in physical consistency over baselines.

Effective enforcement of temporally coherent 4D dynamics.

Maintains high-quality generative performance.

Abstract

Recent video diffusion models have achieved impressive capabilities as large-scale generative world models. However, these models often struggle with fine-grained physical consistency, exhibiting physically implausible dynamics over time. In this work, we present \textbf{Phys4D}, a pipeline for learning physics-consistent 4D world representations from video diffusion models. Phys4D adopts \textbf{a three-stage training paradigm} that progressively lifts appearance-driven video diffusion models into physics-consistent 4D world representations. We first bootstrap robust geometry and motion representations through large-scale pseudo-supervised pretraining, establishing a foundation for 4D scene modeling. We then perform physics-grounded supervised fine-tuning using simulation-generated data, enforcing temporally consistent 4D dynamics. Finally, we apply simulation-grounded reinforcement learning to correct residual physical violations that are difficult to capture through explicit supervision. To evaluate fine-grained physical consistency beyond appearance-based metrics, we introduce a set of \textbf{4D world consistency evaluation} that probe geometric coherence, motion stability, and long-horizon physical plausibility. Experimental results demonstrate that Phys4D substantially improves fine-grained spatiotemporal and physical consistency compared to appearance-driven baselines, while maintaining strong generative performance. Our project page is available at https://sensational-brioche-7657e7.netlify.app/