GaussianProperty: Integrating Physical Properties to 3D Gaussians with LMMs

TL;DR

GaussianProperty introduces a training-free framework that assigns physical material properties to 3D Gaussians from multi-view images, enabling applications in physics simulation and robotic grasping.

Contribution

It integrates SAM and GPT-4V(ision) for physical property reasoning and projects these properties onto 3D Gaussians, advancing physical understanding from visual data.

Findings

Effective in material segmentation and physical property estimation.

Enhances physics-based dynamic simulation accuracy.

Improves robotic grasping safety and reliability.

Abstract

Estimating physical properties for visual data is a crucial task in computer vision, graphics, and robotics, underpinning applications such as augmented reality, physical simulation, and robotic grasping. However, this area remains under-explored due to the inherent ambiguities in physical property estimation. To address these challenges, we introduce GaussianProperty, a training-free framework that assigns physical properties of materials to 3D Gaussians. Specifically, we integrate the segmentation capability of SAM with the recognition capability of GPT-4V(ision) to formulate a global-local physical property reasoning module for 2D images. Then we project the physical properties from multi-view 2D images to 3D Gaussians using a voting strategy. We demonstrate that 3D Gaussians with physical property annotations enable applications in physics-based dynamic simulation and robotic grasping. For physics-based dynamic simulation, we leverage the Material Point Method (MPM) for realistic dynamic simulation. For robot grasping, we develop a grasping force prediction strategy that estimates a safe force range required for object grasping based on the estimated physical properties. Extensive experiments on material segmentation, physics-based dynamic simulation, and robotic grasping validate the effectiveness of our proposed method, highlighting its crucial role in understanding physical properties from visual data. Online demo, code, more cases and annotated datasets are available on \href{https://Gaussian-Property.github.io}{this https URL}.