PhysFlow: Unleashing the Potential of Multi-modal Foundation Models and Video Diffusion for 4D Dynamic Physical Scene Simulation

TL;DR

PhysFlow combines multi-modal foundation models and video diffusion to significantly improve the realism and accuracy of 4D dynamic physical scene simulations, capturing complex material properties and interactions.

Contribution

It introduces a novel framework integrating multi-modal models and video diffusion with differentiable MPM for enhanced physical scene simulation.

Findings

Improved simulation accuracy over existing methods

Enhanced modeling of complex material interactions

Realistic dynamic scene predictions

Abstract

Realistic simulation of dynamic scenes requires accurately capturing diverse material properties and modeling complex object interactions grounded in physical principles. However, existing methods are constrained to basic material types with limited predictable parameters, making them insufficient to represent the complexity of real-world materials. We introduce PhysFlow, a novel approach that leverages multi-modal foundation models and video diffusion to achieve enhanced 4D dynamic scene simulation. Our method utilizes multi-modal models to identify material types and initialize material parameters through image queries, while simultaneously inferring 3D Gaussian splats for detailed scene representation. We further refine these material parameters using video diffusion with a differentiable Material Point Method (MPM) and optical flow guidance rather than render loss or Score Distillation Sampling (SDS) loss. This integrated framework enables accurate prediction and realistic simulation of dynamic interactions in real-world scenarios, advancing both accuracy and flexibility in physics-based simulations.