Global-Local Collaborative Inference with LLM for Lidar-Based Open-Vocabulary Detection

TL;DR

This paper introduces a novel global-local collaborative inference framework using LLMs for lidar-based open-vocabulary detection, effectively integrating scene and object-level features for improved 3D object detection accuracy.

Contribution

It proposes a global-local scheme (GLIS) that combines scene-level and object-level features with LLM-based reasoning, along with pseudo label generation and background-aware localization for 3D OVD.

Findings

Outperforms existing lidar-based OVD methods on ScanNetV2 and SUN RGB-D datasets.

Effectively integrates scene and object features for improved detection.

Demonstrates the effectiveness of LLM-based chain-of-thought inference in 3D detection.

Abstract

Open-Vocabulary Detection (OVD) is the task of detecting all interesting objects in a given scene without predefined object classes. Extensive work has been done to deal with the OVD for 2D RGB images, but the exploration of 3D OVD is still limited. Intuitively, lidar point clouds provide 3D information, both object level and scene level, to generate trustful detection results. However, previous lidar-based OVD methods only focus on the usage of object-level features, ignoring the essence of scene-level information. In this paper, we propose a Global-Local Collaborative Scheme (GLIS) for the lidar-based OVD task, which contains a local branch to generate object-level detection result and a global branch to obtain scene-level global feature. With the global-local information, a Large Language Model (LLM) is applied for chain-of-thought inference, and the detection result can be refined accordingly. We further propose Reflected Pseudo Labels Generation (RPLG) to generate high-quality pseudo labels for supervision and Background-Aware Object Localization (BAOL) to select precise object proposals. Extensive experiments on ScanNetV2 and SUN RGB-D demonstrate the superiority of our methods. Code is released at https://github.com/GradiusTwinbee/GLIS.