Lifting by Gaussians: A Simple, Fast and Flexible Method for 3D Instance Segmentation

TL;DR

Lifting By Gaussians (LBG) is a fast, flexible, and scene-agnostic method for 3D instance segmentation that leverages 2D segmentation masks and features to directly segment 3D Gaussian radiance fields without per-scene training.

Contribution

LBG introduces a novel, no-training-needed approach for 3D instance segmentation that is faster, simpler, and more modular than previous methods, enabling high-quality semantic segmentation of 3DGS fields.

Findings

LBG operates seamlessly on existing 3DGS reconstructions without per-scene training.

LBG achieves superior semantic segmentation for novel view synthesis and asset extraction.

LBG is an order of magnitude faster and simpler than previous approaches.

Abstract

We introduce Lifting By Gaussians (LBG), a novel approach for open-world instance segmentation of 3D Gaussian Splatted Radiance Fields (3DGS). Recently, 3DGS Fields have emerged as a highly efficient and explicit alternative to Neural Field-based methods for high-quality Novel View Synthesis. Our 3D instance segmentation method directly lifts 2D segmentation masks from SAM (alternately FastSAM, etc.), together with features from CLIP and DINOv2, directly fusing them onto 3DGS (or similar Gaussian radiance fields such as 2DGS). Unlike previous approaches, LBG requires no per-scene training, allowing it to operate seamlessly on any existing 3DGS reconstruction. Our approach is not only an order of magnitude faster and simpler than existing approaches; it is also highly modular, enabling 3D semantic segmentation of existing 3DGS fields without requiring a specific parametrization of the 3D Gaussians. Furthermore, our technique achieves superior semantic segmentation for 2D semantic novel view synthesis and 3D asset extraction results while maintaining flexibility and efficiency. We further introduce a novel approach to evaluate individually segmented 3D assets from 3D radiance field segmentation methods.