Gaussian Grouping: Segment and Edit Anything in 3D Scenes
Mingqiao Ye, Martin Danelljan, Fisher Yu, Lei Ke

TL;DR
Gaussian Grouping extends Gaussian Splatting to enable joint 3D scene reconstruction and fine-grained object segmentation, leveraging 2D mask supervision and enabling versatile scene editing with high quality and efficiency.
Contribution
It introduces Gaussian Grouping with Identity Encodings for object-aware 3D scene understanding and editing, using 2D supervision without expensive 3D labels.
Findings
Achieves high-quality 3D reconstruction and segmentation.
Enables versatile scene editing tasks like object removal and colorization.
Demonstrates efficiency and fine granularity in 3D scene editing.
Abstract
The recent Gaussian Splatting achieves high-quality and real-time novel-view synthesis of the 3D scenes. However, it is solely concentrated on the appearance and geometry modeling, while lacking in fine-grained object-level scene understanding. To address this issue, we propose Gaussian Grouping, which extends Gaussian Splatting to jointly reconstruct and segment anything in open-world 3D scenes. We augment each Gaussian with a compact Identity Encoding, allowing the Gaussians to be grouped according to their object instance or stuff membership in the 3D scene. Instead of resorting to expensive 3D labels, we supervise the Identity Encodings during the differentiable rendering by leveraging the 2D mask predictions by Segment Anything Model (SAM), along with introduced 3D spatial consistency regularization. Compared to the implicit NeRF representation, we show that the discrete and…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Vision and Imaging · 3D Shape Modeling and Analysis · Computer Graphics and Visualization Techniques
MethodsSegment Anything Model · Colorization
