Open Vocabulary Monocular 3D Object Detection

TL;DR

This paper introduces open-vocabulary monocular 3D object detection, enabling detection of any object category from a single RGB image without relying on costly sensors or limited vocabularies.

Contribution

It presents a framework that leverages pretrained vision models and a new evaluation metric to improve zero-shot and in-domain 3D detection performance.

Findings

Achieves state-of-the-art zero-shot detection results.

Outperforms existing methods on in-domain detection.

Provides a new benchmark and evaluation protocol.

Abstract

We propose and study open-vocabulary monocular 3D detection, a novel task that aims to detect objects of any categores in metric 3D space from a single RGB image. Existing 3D object detectors either rely on costly sensors such as LiDAR or multi-view setups, or remain confined to closed vocabularies settings with limited categories, restricting their applicability. We identify two key challenges in this new setting. First, the scarcity of 3D bounding box annotations limits the ability to train generalizable models. To reduce dependence on 3D supervision, we propose a framework that effectively integrates pretrained 2D and 3D vision foundation models. Second, missing labels and semantic ambiguities (\eg, table vs. desk) in existing datasets hinder reliable evaluation. To address this, we design a novel metric that captures model performance while mitigating annotation issues. Our approach achieves state-of-the-art results in zero-shot 3D detection of novel categories as well as in-domain detection on seen classes. We hope our method provides a strong baseline and our evaluation protocol establishes a reliable benchmark for future research.