LabelAny3D: Label Any Object 3D in the Wild
Jin Yao, Radowan Mahmud Redoy, Sebastian Elbaum, Matthew B. Dwyer, Zezhou Cheng

TL;DR
LabelAny3D introduces an analysis-by-synthesis framework for generating high-quality 3D annotations from 2D images, enabling improved monocular 3D detection in diverse, real-world scenarios.
Contribution
It presents a novel analysis-by-synthesis approach and a new COCO3D benchmark for open-vocabulary monocular 3D detection, addressing data scarcity and annotation challenges.
Findings
Annotations improve 3D detection accuracy across benchmarks.
Outperforms prior auto-labeling methods in quality.
Demonstrates potential of foundation-model-driven annotation.
Abstract
Detecting objects in 3D space from monocular input is crucial for applications ranging from robotics to scene understanding. Despite advanced performance in the indoor and autonomous driving domains, existing monocular 3D detection models struggle with in-the-wild images due to the lack of 3D in-the-wild datasets and the challenges of 3D annotation. We introduce LabelAny3D, an \emph{analysis-by-synthesis} framework that reconstructs holistic 3D scenes from 2D images to efficiently produce high-quality 3D bounding box annotations. Built on this pipeline, we present COCO3D, a new benchmark for open-vocabulary monocular 3D detection, derived from the MS-COCO dataset and covering a wide range of object categories absent from existing 3D datasets. Experiments show that annotations generated by LabelAny3D improve monocular 3D detection performance across multiple benchmarks, outperforming…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
Taxonomy
TopicsAdvanced Neural Network Applications · 3D Shape Modeling and Analysis · Robotics and Sensor-Based Localization
