3D-AVS: LiDAR-based 3D Auto-Vocabulary Segmentation

TL;DR

3D-AVS introduces an auto-vocabulary segmentation method for 3D point clouds that generates semantic categories at runtime, eliminating the need for human-provided labels and enabling richer, scalable annotations.

Contribution

It proposes a novel auto-vocabulary segmentation approach for 3D point clouds that combines image and LiDAR data, with a new metric for evaluating unknown vocabularies.

Findings

Effective segmentation on nuScenes and ScanNet200 datasets.

Generates accurate semantic classes without human labels.

Enhances robustness under challenging lighting conditions.

Abstract

Open-Vocabulary Segmentation (OVS) methods offer promising capabilities in detecting unseen object categories, but the category must be known and needs to be provided by a human, either via a text prompt or pre-labeled datasets, thus limiting their scalability. We propose 3D-AVS, a method for Auto-Vocabulary Segmentation of 3D point clouds for which the vocabulary is unknown and auto-generated for each input at runtime, thus eliminating the human in the loop and typically providing a substantially larger vocabulary for richer annotations. 3D-AVS first recognizes semantic entities from image or point cloud data and then segments all points with the automatically generated vocabulary. Our method incorporates both image-based and point-based recognition, enhancing robustness under challenging lighting conditions where geometric information from LiDAR is especially valuable. Our point-based recognition features a Sparse Masked Attention Pooling (SMAP) module to enrich the diversity of recognized objects. To address the challenges of evaluating unknown vocabularies and avoid annotation biases from label synonyms, hierarchies, or semantic overlaps, we introduce the annotation-free Text-Point Semantic Similarity (TPSS) metric for assessing generated vocabulary quality. Our evaluations on nuScenes and ScanNet200 demonstrate 3D-AVS's ability to generate semantic classes with accurate point-wise segmentations. Codes will be released at https://github.com/ozzyou/3D-AVS