PhysMotion: Physics-Grounded Dynamics From a Single Image

TL;DR

PhysMotion introduces a physics-based simulation framework that generates realistic, physically plausible videos from a single image by combining continuum mechanics, differentiable simulation, and diffusion-based refinement.

Contribution

It is the first to integrate continuum mechanics-based simulations with deep generative models for high-quality, physics-consistent video synthesis from a single image.

Findings

Produces physically plausible videos with detailed dynamics.

Outperforms traditional data-driven models in motion consistency.

Effectively combines simulation with diffusion models for detail enhancement.

Abstract

We introduce PhysMotion, a novel framework that leverages principled physics-based simulations to guide intermediate 3D representations generated from a single image and input conditions (e.g., applied force and torque), producing high-quality, physically plausible video generation. By utilizing continuum mechanics-based simulations as a prior knowledge, our approach addresses the limitations of traditional data-driven generative models and result in more consistent physically plausible motions. Our framework begins by reconstructing a feed-forward 3D Gaussian from a single image through geometry optimization. This representation is then time-stepped using a differentiable Material Point Method (MPM) with continuum mechanics-based elastoplasticity models, which provides a strong foundation for realistic dynamics, albeit at a coarse level of detail. To enhance the geometry, appearance and ensure spatiotemporal consistency, we refine the initial simulation using a text-to-image (T2I) diffusion model with cross-frame attention, resulting in a physically plausible video that retains intricate details comparable to the input image. We conduct comprehensive qualitative and quantitative evaluations to validate the efficacy of our method. Our project page is available at: https://supertan0204.github.io/physmotion_website/.