InstanceGaussian: Appearance-Semantic Joint Gaussian Representation for 3D Instance-Level Perception

TL;DR

InstanceGaussian introduces a joint appearance-semantic Gaussian representation for 3D scene understanding, effectively balancing and integrating features to improve instance segmentation accuracy in complex 3D environments.

Contribution

It proposes a novel joint appearance-semantic Gaussian representation, a progressive training strategy, and a bottom-up instance aggregation method for enhanced 3D scene segmentation.

Findings

Achieves state-of-the-art results in open-vocabulary 3D point-level segmentation.

Effectively balances appearance and semantic features for better boundary delineation.

Addresses segmentation challenges with a category-agnostic, bottom-up approach.

Abstract

3D scene understanding has become an essential area of research with applications in autonomous driving, robotics, and augmented reality. Recently, 3D Gaussian Splatting (3DGS) has emerged as a powerful approach, combining explicit modeling with neural adaptability to provide efficient and detailed scene representations. However, three major challenges remain in leveraging 3DGS for scene understanding: 1) an imbalance between appearance and semantics, where dense Gaussian usage for fine-grained texture modeling does not align with the minimal requirements for semantic attributes; 2) inconsistencies between appearance and semantics, as purely appearance-based Gaussians often misrepresent object boundaries; and 3) reliance on top-down instance segmentation methods, which struggle with uneven category distributions, leading to over- or under-segmentation. In this work, we propose InstanceGaussian, a method that jointly learns appearance and semantic features while adaptively aggregating instances. Our contributions include: i) a novel Semantic-Scaffold-GS representation balancing appearance and semantics to improve feature representations and boundary delineation; ii) a progressive appearance-semantic joint training strategy to enhance stability and segmentation accuracy; and iii) a bottom-up, category-agnostic instance aggregation approach that addresses segmentation challenges through farthest point sampling and connected component analysis. Our approach achieves state-of-the-art performance in category-agnostic, open-vocabulary 3D point-level segmentation, highlighting the effectiveness of the proposed representation and training strategies. Project page: https://lhj-git.github.io/InstanceGaussian/