VoteSplat: Hough Voting Gaussian Splatting for 3D Scene Understanding

TL;DR

VoteSplat introduces a novel framework combining Hough voting with 3D Gaussian Splatting, enhancing 3D scene understanding and object localization while reducing training costs in high-dimensional semantic spaces.

Contribution

It integrates Hough voting with 3D Gaussian Splatting and utilizes SAM for instance segmentation, enabling open-vocabulary 3D object localization with lower training costs.

Findings

Effective open-vocabulary 3D instance localization

Improved 3D point cloud understanding

Reduced training costs for semantic mapping

Abstract

3D Gaussian Splatting (3DGS) has become horsepower in high-quality, real-time rendering for novel view synthesis of 3D scenes. However, existing methods focus primarily on geometric and appearance modeling, lacking deeper scene understanding while also incurring high training costs that complicate the originally streamlined differentiable rendering pipeline. To this end, we propose VoteSplat, a novel 3D scene understanding framework that integrates Hough voting with 3DGS. Specifically, Segment Anything Model (SAM) is utilized for instance segmentation, extracting objects, and generating 2D vote maps. We then embed spatial offset vectors into Gaussian primitives. These offsets construct 3D spatial votes by associating them with 2D image votes, while depth distortion constraints refine localization along the depth axis. For open-vocabulary object localization, VoteSplat maps 2D image semantics to 3D point clouds via voting points, reducing training costs associated with high-dimensional CLIP features while preserving semantic unambiguity. Extensive experiments demonstrate effectiveness of VoteSplat in open-vocabulary 3D instance localization, 3D point cloud understanding, click-based 3D object localization, hierarchical segmentation, and ablation studies. Our code is available at https://sy-ja.github.io/votesplat/