Trace3D: Consistent Segmentation Lifting via Gaussian Instance Tracing

TL;DR

Trace3D introduces Gaussian Instance Tracing, a novel method that enhances 3D segmentation consistency and boundary sharpness by leveraging Gaussian representations and instance-aware refinement, improving 3D asset extraction and scene editing.

Contribution

The paper proposes Gaussian Instance Tracing (GIT), a new approach that refines 3D segmentation by correcting 2D inconsistencies and adaptively splitting Gaussians, advancing the state-of-the-art in 3D segmentation lifting.

Findings

Improves 3D segmentation consistency and boundary sharpness.

Enables better 3D asset extraction and scene editing.

Demonstrates effectiveness in online and offline settings.

Abstract

We address the challenge of lifting 2D visual segmentation to 3D in Gaussian Splatting. Existing methods often suffer from inconsistent 2D masks across viewpoints and produce noisy segmentation boundaries as they neglect these semantic cues to refine the learned Gaussians. To overcome this, we introduce Gaussian Instance Tracing (GIT), which augments the standard Gaussian representation with an instance weight matrix across input views. Leveraging the inherent consistency of Gaussians in 3D, we use this matrix to identify and correct 2D segmentation inconsistencies. Furthermore, since each Gaussian ideally corresponds to a single object, we propose a GIT-guided adaptive density control mechanism to split and prune ambiguous Gaussians during training, resulting in sharper and more coherent 2D and 3D segmentation boundaries. Experimental results show that our method extracts clean 3D assets and consistently improves 3D segmentation in both online (e.g., self-prompting) and offline (e.g., contrastive lifting) settings, enabling applications such as hierarchical segmentation, object extraction, and scene editing.