3D-MOOD: Lifting 2D to 3D for Monocular Open-Set Object Detection

TL;DR

This paper introduces 3D-MOOD, an end-to-end monocular 3D object detection method capable of handling open-set scenarios with novel objects and environments, achieving state-of-the-art results across multiple datasets.

Contribution

The paper presents the first open-set monocular 3D detection framework that lifts 2D detections into 3D space and incorporates geometry priors for better generalization.

Findings

Achieves state-of-the-art results on Omni3D, Argoverse 2, and ScanNet datasets.

Effectively generalizes to unseen object categories and environments.

Demonstrates the benefit of joint 2D-3D training and canonical image space for open-set detection.

Abstract

Monocular 3D object detection is valuable for various applications such as robotics and AR/VR. Existing methods are confined to closed-set settings, where the training and testing sets consist of the same scenes and/or object categories. However, real-world applications often introduce new environments and novel object categories, posing a challenge to these methods. In this paper, we address monocular 3D object detection in an open-set setting and introduce the first end-to-end 3D Monocular Open-set Object Detector (3D-MOOD). We propose to lift the open-set 2D detection into 3D space through our designed 3D bounding box head, enabling end-to-end joint training for both 2D and 3D tasks to yield better overall performance. We condition the object queries with geometry prior and overcome the generalization for 3D estimation across diverse scenes. To further improve performance, we design the canonical image space for more efficient cross-dataset training. We evaluate 3D-MOOD on both closed-set settings (Omni3D) and open-set settings (Omni3D to Argoverse 2, ScanNet), and achieve new state-of-the-art results. Code and models are available at royyang0714.github.io/3D-MOOD.