Split&Splat: Zero-Shot Panoptic Segmentation via Explicit Instance Modeling and 3D Gaussian Splatting

TL;DR

Split&Splat introduces a novel framework for panoptic scene reconstruction that explicitly models object instances and integrates semantic descriptors, enabling high-quality, view-consistent 3D segmentation and editing.

Contribution

It is the first method to combine explicit instance modeling with 3D Gaussian Splatting for panoptic scene reconstruction, improving downstream task performance.

Findings

Achieves state-of-the-art results on ScanNetv2 benchmark.

Supports various applications like object retrieval and 3D editing.

Provides view-consistent, object-aware 3D reconstructions.

Abstract

3D Gaussian Splatting (GS) enables fast and high-quality scene reconstruction, but it lacks an object-consistent and semantically aware structure. We propose Split&Splat, a framework for panoptic scene reconstruction using 3DGS. Our approach explicitly models object instances. It first propagates instance masks across views using depth, thus producing view-consistent 2D masks. Each object is then reconstructed independently and merged back into the scene while refining its boundaries. Finally, instance-level semantic descriptors are embedded in the reconstructed objects, supporting various applications, including panoptic segmentation, object retrieval, and 3D editing. Unlike existing methods, Split&Splat tackles the problem by first segmenting the scene and then reconstructing each object individually. This design naturally supports downstream tasks and allows Split&Splat to achieve state-of-the-art performance on the ScanNetv2 segmentation benchmark.