MaGS: Reconstructing and Simulating Dynamic 3D Objects with Mesh-adsorbed Gaussian Splatting

TL;DR

MaGS introduces a novel mesh-adsorbed Gaussian splatting method that combines the flexibility of Gaussian representations with structured mesh constraints, enabling improved 3D reconstruction and dynamic simulation from video data.

Contribution

The paper proposes MaGS, a new method that integrates mesh and Gaussian representations for 3D reconstruction and simulation, along with RMD-Net, RGD-Net, and MPE-Net for learning motion priors and deformation modeling.

Findings

Achieves state-of-the-art results on multiple datasets.

Effectively models complex deformations beyond input videos.

Compatible with various deformation priors like ARAP, SMPL, and soft physics.

Abstract

3D reconstruction and simulation, although interrelated, have distinct objectives: reconstruction requires a flexible 3D representation that can adapt to diverse scenes, while simulation needs a structured representation to model motion principles effectively. This paper introduces the Mesh-adsorbed Gaussian Splatting (MaGS) method to address this challenge. MaGS constrains 3D Gaussians to roam near the mesh, creating a mutually adsorbed mesh-Gaussian 3D representation. Such representation harnesses both the rendering flexibility of 3D Gaussians and the structured property of meshes. To achieve this, we introduce RMD-Net, a network that learns motion priors from video data to refine mesh deformations, alongside RGD-Net, which models the relative displacement between the mesh and Gaussians to enhance rendering fidelity under mesh constraints. To generalize to novel, user-defined deformations beyond input video without reliance on temporal data, we propose MPE-Net, which leverages inherent mesh information to bootstrap RMD-Net and RGD-Net. Due to the universality of meshes, MaGS is compatible with various deformation priors such as ARAP, SMPL, and soft physics simulation. Extensive experiments on the D-NeRF, DG-Mesh, and PeopleSnapshot datasets demonstrate that MaGS achieves state-of-the-art performance in both reconstruction and simulation.