PointGauss: Point Cloud-Guided Multi-Object Segmentation for Gaussian Splatting

TL;DR

PointGauss introduces a real-time, point cloud-guided multi-object segmentation framework for Gaussian Splatting that improves efficiency and multi-view consistency, supported by a new comprehensive 3D segmentation dataset.

Contribution

The paper presents a novel point cloud-guided segmentation method for Gaussian Splatting and introduces DesktopObjects-360, a large-scale 3D segmentation dataset with comprehensive annotations.

Findings

Achieves 1.89 to 31.78% improvement in multi-view mIoU.

Generates 3D instance masks within 1 minute.

Demonstrates superior computational efficiency and multi-view consistency.

Abstract

We introduce PointGauss, a novel point cloud-guided framework for real-time multi-object segmentation in Gaussian Splatting representations. Unlike existing methods that suffer from prolonged initialization and limited multi-view consistency, our approach achieves efficient 3D segmentation by directly parsing Gaussian primitives through a point cloud segmentation-driven pipeline. The key innovation lies in two aspects: (1) a point cloud-based Gaussian primitive decoder that generates 3D instance masks within 1 minute, and (2) a GPU-accelerated 2D mask rendering system that ensures multi-view consistency. Extensive experiments demonstrate significant improvements over previous state-of-the-art methods, achieving performance gains of 1.89 to 31.78% in multi-view mIoU, while maintaining superior computational efficiency. To address the limitations of current benchmarks (single-object focus, inconsistent 3D evaluation, small scale, and partial coverage), we present DesktopObjects-360, a novel comprehensive dataset for 3D segmentation in radiance fields, featuring: (1) complex multi-object scenes, (2) globally consistent 2D annotations, (3) large-scale training data (over 27 thousand 2D masks), (4) full 360{\deg} coverage, and (5) 3D evaluation masks.