DreamPhysics: Learning Physics-Based 3D Dynamics with Video Diffusion Priors

TL;DR

This paper introduces DreamPhysics, a method that combines video diffusion priors with physics-based simulation to generate realistic 4D dynamic content of 3D scenes, overcoming limitations of previous approaches.

Contribution

It proposes a novel approach to learn physical properties of materials from videos and use a physics-based simulator for realistic 4D content creation.

Findings

Produces more realistic 4D motions than existing methods

Effectively integrates video diffusion priors with physics simulation

Demonstrates improved motion realism in experimental results

Abstract

Dynamic 3D interaction has been attracting a lot of attention recently. However, creating such 4D content remains challenging. One solution is to animate 3D scenes with physics-based simulation, which requires manually assigning precise physical properties to the object or the simulated results would become unnatural. Another solution is to learn the deformation of 3D objects with the distillation of video generative models, which, however, tends to produce 3D videos with small and discontinuous motions due to the inappropriate extraction and application of physics priors. In this work, to combine the strengths and complementing shortcomings of the above two solutions, we propose to learn the physical properties of a material field with video diffusion priors, and then utilize a physics-based Material-Point-Method (MPM) simulator to generate 4D content with realistic motions. In particular, we propose motion distillation sampling to emphasize video motion information during distillation. In addition, to facilitate the optimization, we further propose a KAN-based material field with frame boosting. Experimental results demonstrate that our method enjoys more realistic motions than state-of-the-arts do.