SplatMesh: Interactive 3D Segmentation and Editing Using Mesh-Based Gaussian Splatting

TL;DR

SplatMesh is a novel method that combines mesh simplification with Gaussian splatting for efficient, high-quality 3D segmentation and editing, balancing detail, stability, and memory constraints.

Contribution

It introduces a new algorithm that integrates mesh simplification with Gaussian splatting, enabling flexible, high-quality 3D editing under memory constraints.

Findings

Outperforms existing methods in view synthesis quality.

Enables extensive 3D editing with improved stability.

Balances memory use with detailed representation.

Abstract

A key challenge in fine-grained 3D-based interactive editing is the absence of an efficient representation that balances diverse modifications with high-quality view synthesis under a given memory constraint. While 3D meshes provide robustness for various modifications, they often yield lower-quality view synthesis compared to 3D Gaussian Splatting, which, in turn, suffers from instability during extensive editing. A straightforward combination of these two representations results in suboptimal performance and fails to meet memory constraints. In this paper, we introduce SplatMesh, a novel fine-grained interactive 3D segmentation and editing algorithm that integrates 3D Gaussian Splat with a precomputed mesh and could adjust the memory request based on the requirement. Specifically, given a mesh, \method simplifies it while considering both color and shape, ensuring it meets memory constraints. Then, SplatMesh aligns Gaussian splats with the simplified mesh by treating each triangle as a new reference point. By segmenting and editing the simplified mesh, we can effectively edit the Gaussian splats as well, which will lead to extensive experiments on real and synthetic datasets, coupled with illustrative visual examples, highlighting the superiority of our approach in terms of representation quality and editing performance. Code of our paper can be found here: https://github.com/kaichen-z/SplatMesh.