PIG: Physically-based Multi-Material Interaction with 3D Gaussians

TL;DR

PIG introduces a physically-based multi-material interaction framework using 3D Gaussians, enabling precise segmentation, realistic deformation, and artifact-free rendering for dynamic scene reconstruction.

Contribution

It combines 3D object segmentation with physical property assignment and deformation constraints to improve realism and accuracy in 3D scene modeling with Gaussians.

Findings

Outperforms SOTA in visual quality

Enables accurate multi-material interactions

Reduces rendering artifacts

Abstract

3D Gaussian Splatting has achieved remarkable success in reconstructing both static and dynamic 3D scenes. However, in a scene represented by 3D Gaussian primitives, interactions between objects suffer from inaccurate 3D segmentation, imprecise deformation among different materials, and severe rendering artifacts. To address these challenges, we introduce PIG: Physically-Based Multi-Material Interaction with 3D Gaussians, a novel approach that combines 3D object segmentation with the simulation of interacting objects in high precision. Firstly, our method facilitates fast and accurate mapping from 2D pixels to 3D Gaussians, enabling precise 3D object-level segmentation. Secondly, we assign unique physical properties to correspondingly segmented objects within the scene for multi-material coupled interactions. Finally, we have successfully embedded constraint scales into deformation gradients, specifically clamping the scaling and rotation properties of the Gaussian primitives to eliminate artifacts and achieve geometric fidelity and visual consistency. Experimental results demonstrate that our method not only outperforms the state-of-the-art (SOTA) in terms of visual quality, but also opens up new directions and pipelines for the field of physically realistic scene generation.