Auto-Vocabulary 3D Object Detection

TL;DR

This paper introduces AV3DOD, a novel framework for automatic vocabulary generation in 3D object detection using vision-language models, achieving state-of-the-art results without user-specified class labels.

Contribution

The paper presents a new method that automatically generates class labels for 3D objects, eliminating the need for user input and improving detection and semantic quality.

Findings

Achieves state-of-the-art localization (mAP) on ScanNetV2 and SUNRGB-D datasets.

Surpasses previous methods like CoDA by 3.48 mAP on ScanNetV2.

Attains a 24.5% relative improvement in semantic score (SS).

Abstract

Open-vocabulary 3D object detection methods are able to localize 3D boxes of classes unseen during training. Despite the name, existing methods rely on user-specified classes both at training and inference. We propose to study Auto-Vocabulary 3D Object Detection (AV3DOD), where the classes are automatically generated for the detected objects without any user input. To this end, we introduce Semantic Score (SS) to evaluate the quality of the generated class names. We then develop a novel framework, AV3DOD, which leverages 2D vision-language models (VLMs) to generate rich semantic candidates through image captioning, pseudo 3D box generation, and feature-space semantics expansion. AV3DOD achieves the state-of-the-art (SOTA) performance on both localization (mAP) and semantic quality (SS) on the ScanNetV2 and SUNRGB-D datasets. Notably, it surpasses the SOTA, CoDA, by 3.48 overall mAP and attains a 24.5% relative improvement in SS on ScanNetV2.