Efficient Physics Simulation for 3D Scenes via MLLM-Guided Gaussian Splatting

TL;DR

This paper introduces Sim Anything, a novel physics simulation framework that leverages multi-modal large language models to predict physical properties and simulate dynamic 3D scenes efficiently, achieving realistic motion with reduced computational costs.

Contribution

The paper presents a new approach combining MLLM-based property perception and probabilistic distribution estimation for efficient physics simulation of 3D scenes.

Findings

Achieves realistic 3D object motion within 2 minutes on a single GPU.

Reduces computational costs compared to existing methods.

Outperforms state-of-the-art in realism of simulated dynamics.

Abstract

Recent advancements in 3D generation models have opened new possibilities for simulating dynamic 3D object movements and customizing behaviors, yet creating this content remains challenging. Current methods often require manual assignment of precise physical properties for simulations or rely on video generation models to predict them, which is computationally intensive. In this paper, we rethink the usage of multi-modal large language model (MLLM) in physics-based simulation, and present Sim Anything, a physics-based approach that endows static 3D objects with interactive dynamics. We begin with detailed scene reconstruction and object-level 3D open-vocabulary segmentation, progressing to multi-view image in-painting. Inspired by human visual reasoning, we propose MLLM-based Physical Property Perception (MLLM-P3) to predict mean physical properties of objects in a zero-shot manner. Based on the mean values and the object's geometry, the Material Property Distribution Prediction model (MPDP) model then estimates the full distribution, reformulating the problem as probability distribution estimation to reduce computational costs. Finally, we simulate objects in an open-world scene with particles sampled via the Physical-Geometric Adaptive Sampling (PGAS) strategy, efficiently capturing complex deformations and significantly reducing computational costs. Extensive experiments and user studies demonstrate our Sim Anything achieves more realistic motion than state-of-the-art methods within 2 minutes on a single GPU.