MotionPhysics: Learnable Motion Distillation for Text-Guided Simulation

TL;DR

MotionPhysics introduces a differentiable framework that infers physical parameters from natural language prompts to produce realistic 3D object simulations without ground-truth data, leveraging large language models and pretrained video diffusion models.

Contribution

It presents a novel end-to-end method combining language models and learnable motion distillation to automatically generate physically plausible parameters for 3D simulations from natural language.

Findings

Outperforms state-of-the-art in realistic 3D simulation quality.

Works across diverse materials and object types.

Automatically infers plausible physical parameters from language prompts.

Abstract

Accurately simulating existing 3D objects and a wide variety of materials often demands expert knowledge and time-consuming physical parameter tuning to achieve the desired dynamic behavior. We introduce MotionPhysics, an end-to-end differentiable framework that infers plausible physical parameters from a user-provided natural language prompt for a chosen 3D scene of interest, removing the need for guidance from ground-truth trajectories or annotated videos. Our approach first utilizes a multimodal large language model to estimate material parameter values, which are constrained to lie within plausible ranges. We further propose a learnable motion distillation loss that extracts robust motion priors from pretrained video diffusion models while minimizing appearance and geometry inductive biases to guide the simulation. We evaluate MotionPhysics across more than thirty scenarios, including real-world, human-designed, and AI-generated 3D objects, spanning a wide range of materials such as elastic solids, metals, foams, sand, and both Newtonian and non-Newtonian fluids. We demonstrate that MotionPhysics produces visually realistic dynamic simulations guided by natural language, surpassing the state of the art while automatically determining physically plausible parameters. The code and project page are available at: https://wangmiaowei.github.io/MotionPhysics.github.io/.