VideoPhy: Evaluating Physical Commonsense for Video Generation

TL;DR

VideoPhy introduces a benchmark to evaluate whether current text-to-video models generate videos that adhere to physical commonsense, revealing significant gaps in their ability to simulate real-world physics.

Contribution

The paper presents VideoPhy, a new benchmark and auto-evaluator for assessing physical commonsense in video generation models, highlighting their current limitations.

Findings

Existing models often fail to generate physically plausible videos.

The best model, CogVideoX-5B, adheres to physics in only 39.6% of cases.

VideoPhy exposes the gap between current capabilities and real-world physical understanding.

Abstract

Recent advances in internet-scale video data pretraining have led to the development of text-to-video generative models that can create high-quality videos across a broad range of visual concepts, synthesize realistic motions and render complex objects. Hence, these generative models have the potential to become general-purpose simulators of the physical world. However, it is unclear how far we are from this goal with the existing text-to-video generative models. To this end, we present VideoPhy, a benchmark designed to assess whether the generated videos follow physical commonsense for real-world activities (e.g. marbles will roll down when placed on a slanted surface). Specifically, we curate diverse prompts that involve interactions between various material types in the physical world (e.g., solid-solid, solid-fluid, fluid-fluid). We then generate videos conditioned on these captions from diverse state-of-the-art text-to-video generative models, including open models (e.g., CogVideoX) and closed models (e.g., Lumiere, Dream Machine). Our human evaluation reveals that the existing models severely lack the ability to generate videos adhering to the given text prompts, while also lack physical commonsense. Specifically, the best performing model, CogVideoX-5B, generates videos that adhere to the caption and physical laws for 39.6% of the instances. VideoPhy thus highlights that the video generative models are far from accurately simulating the physical world. Finally, we propose an auto-evaluator, VideoCon-Physics, to assess the performance reliably for the newly released models.