Towards Camera Open-set 3D Object Detection for Autonomous Driving Scenarios

TL;DR

This paper introduces OS-Det3D, a two-stage framework that enables camera-based 3D object detectors to recognize both known and unknown objects in autonomous driving, improving safety and detection robustness.

Contribution

The paper proposes a novel open-set 3D detection framework using geometric cues and a joint selection module, allowing detection of unseen objects beyond predefined categories.

Findings

Enhanced detection of unknown objects in autonomous driving scenarios.

Improved performance on known object categories.

Effective filtering of noisy proposals in cluttered scenes.

Abstract

Conventional camera-based 3D object detectors in autonomous driving are limited to recognizing a predefined set of objects, which poses a safety risk when encountering novel or unseen objects in real-world scenarios. To address this limitation, we present OS-Det3D, a two-stage training framework designed for camera-based open-set 3D object detection. In the first stage, our proposed 3D object discovery network (ODN3D) uses geometric cues from LiDAR point clouds to generate class-agnostic 3D object proposals, each of which are assigned a 3D objectness score. This approach allows the network to discover objects beyond known categories, allowing for the detection of unfamiliar objects. However, due to the absence of class constraints, ODN3D-generated proposals may include noisy data, particularly in cluttered or dynamic scenes. To mitigate this issue, we introduce a joint selection (JS) module in the second stage. The JS module uses both camera bird's eye view (BEV) feature responses and 3D objectness scores to filter out low-quality proposals, yielding high-quality pseudo ground truth for unknown objects. OS-Det3D significantly enhances the ability of camera 3D detectors to discover and identify unknown objects while also improving the performance on known objects, as demonstrated through extensive experiments on the nuScenes and KITTI datasets.