PhysInOne: Visual Physics Learning and Reasoning in One Suite

TL;DR

PhysInOne is a large-scale synthetic dataset with 2 million videos of complex physical phenomena, designed to improve AI understanding and reasoning of physics through extensive annotations and diverse scenes.

Contribution

It introduces the largest physics dataset to date, enabling significant advancements in physics-aware AI tasks and exposing current modeling gaps.

Findings

Fine-tuning models on PhysInOne improves physical plausibility.

The dataset enhances physics-aware video generation and property estimation.

It reveals gaps in modeling complex physical dynamics.

Abstract

We present PhysInOne, a large-scale synthetic dataset addressing the critical scarcity of physically-grounded training data for AI systems. Unlike existing datasets limited to merely hundreds or thousands of examples, PhysInOne provides 2 million videos across 153,810 dynamic 3D scenes, covering 71 basic physical phenomena in mechanics, optics, fluid dynamics, and magnetism. Distinct from previous works, our scenes feature multiobject interactions against complex backgrounds, with comprehensive ground-truth annotations including 3D geometry, semantics, dynamic motion, physical properties, and text descriptions. We demonstrate PhysInOne's efficacy across four emerging applications: physics-aware video generation, long-/short-term future frame prediction, physical property estimation, and motion transfer. Experiments show that fine-tuning foundation models on PhysInOne significantly enhances physical plausibility, while also exposing critical gaps in modeling complex physical dynamics and estimating intrinsic properties. As the largest dataset of its kind, orders of magnitude beyond prior works, PhysInOne establishes a new benchmark for advancing physics-grounded world models in generation, simulation, and embodied AI.