OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understanding

TL;DR

OpenGaussian introduces a novel 3D Gaussian-based approach for point-level open vocabulary understanding, improving feature expressiveness and 3D-2D feature association for robust 3D scene comprehension.

Contribution

It proposes a two-stage feature discretization and an instance-level 3D-2D feature association method for enhanced 3D point-level open vocabulary understanding.

Findings

Effective 3D object selection and understanding demonstrated

Improved 3D-2D feature association accuracy

Robustness in open vocabulary 3D scene understanding

Abstract

This paper introduces OpenGaussian, a method based on 3D Gaussian Splatting (3DGS) capable of 3D point-level open vocabulary understanding. Our primary motivation stems from observing that existing 3DGS-based open vocabulary methods mainly focus on 2D pixel-level parsing. These methods struggle with 3D point-level tasks due to weak feature expressiveness and inaccurate 2D-3D feature associations. To ensure robust feature presentation and 3D point-level understanding, we first employ SAM masks without cross-frame associations to train instance features with 3D consistency. These features exhibit both intra-object consistency and inter-object distinction. Then, we propose a two-stage codebook to discretize these features from coarse to fine levels. At the coarse level, we consider the positional information of 3D points to achieve location-based clustering, which is then refined at the fine level. Finally, we introduce an instance-level 3D-2D feature association method that links 3D points to 2D masks, which are further associated with 2D CLIP features. Extensive experiments, including open vocabulary-based 3D object selection, 3D point cloud understanding, click-based 3D object selection, and ablation studies, demonstrate the effectiveness of our proposed method. The source code is available at our project page: https://3d-aigc.github.io/OpenGaussian