Watching Physics: the Generative Science of Matter and Motion

TL;DR

This paper explores how generative video models, combined with experiments and simulations, can learn and infer physical properties of matter in motion from visual data, advancing scientific understanding.

Contribution

It demonstrates that coupling visual generative models with physics-based data enables scientific inference and highlights the potential of a new Generative Sciences of Matter and Motion.

Findings

Generative models can recover measurable quantities like surface strain in visible kinematics.

Visual plausibility does not always imply physical validity, especially with internal state variables.

Physics-grounded generative models can serve as scientific tools for inference, prediction, and design.

Abstract

Can we learn the physics of matter in motion directly from images and video--and trust it? Answering this question requires integrating experiments, physics-based simulation, and data across traditionally separate disciplines. Much of this knowledge is visual and temporal rather than textual: images and videos encode structure, dynamics, and causality that equations alone cannot fully capture. Recent generative models produce compelling visual content, yet they rely on observational data and often lack physical validity. Here we show that generative video models gain scientific value when they couple visual data with experiments and high-fidelity simulations. Using deformation mechanics as a testbed, we study three systems of increasing complexity--rubber compression, can crushing, and cardiac motion--and identify regimes in which visual learning succeeds, fails, and requires mechanistic supervision. When physics manifests in visible kinematics, generative models recover measurable quantities such as surface strain; when internal state variables dominate, visual plausibility no longer ensures physical admissibility. We propose that this convergence defines a new frontier, the Generative Sciences of Matter and Motion, which unifies Simulogenics, Physiogenics, and Materiogenics. These physics-grounded foundation models can turn visual generation into a scientific instrument for inference, prediction, and design of matter in motion.