MMPhysVideo: Scaling Physical Plausibility in Video Generation via Joint Multimodal Modeling

TL;DR

MMPhysVideo introduces a joint multimodal framework with a novel architecture and data pipeline to enhance physical plausibility in video generation, outperforming existing models.

Contribution

The paper presents MMPhysVideo, a pioneering approach that scales physical plausibility in video diffusion models using joint multimodal modeling and a new data curation pipeline.

Findings

Improves physical plausibility and visual quality in video generation.

Achieves state-of-the-art performance on multiple benchmarks.

Introduces MMPhysPipe for scalable multimodal dataset construction.

Abstract

Despite advancements in generating visually stunning content, video diffusion models (VDMs) often yield physically inconsistent results due to pixel-only reconstruction. To address this, we propose MMPhysVideo, the first framework to scale physical plausibility in video generation through joint multimodal modeling. We recast perceptual cues, specifically semantics, geometry, and spatio-temporal trajectory, into a unified pseudo-RGB format, enabling VDMs to directly capture complex physical dynamics. To mitigate cross-modal interference, we propose a Bidirectionally Controlled Teacher architecture, which utilizes parallel branches to fully decouple RGB and perception processing and adopts two zero-initialized control links to gradually learn pixel-wise consistency. For inference efficiency, the teacher's physical prior is distilled into a single-stream student model via representation alignment. Furthermore, we present MMPhysPipe, a scalable data curation and annotation pipeline tailored for constructing physics-rich multimodal datasets. MMPhysPipe employs a vision-language model (VLM) guided by a chain-of-visual-evidence rule to pinpoint physical subjects, enabling expert models to extract multi-granular perceptual information. Without additional inference costs, MMPhysVideo consistently improves physical plausibility and visual quality over advanced models across various benchmarks and achieves state-of-the-art performance compared to existing methods.