Recursive Cross-View: Use Only 2D Detectors to Achieve 3D Object Detection without 3D Annotations

TL;DR

This paper introduces Recursive Cross-View (RCV), a novel 3D object detection method that relies solely on 2D detectors and no 3D annotations, converting 3D detection into multiple 2D tasks and achieving real-time performance.

Contribution

The proposed RCV method is the first to generate fully oriented 3D bounding boxes without requiring 3D labels, using a recursive paradigm based on 2D detection and cross-view principles.

Findings

Outperforms existing image-based 3D detection methods on SUN RGB-D and KITTI datasets.

Successfully applied to 3D human and hand detection, creating new annotated datasets.

Operates at 7 fps on live RGB-D streams, enabling real-time applications.

Abstract

Heavily relying on 3D annotations limits the real-world application of 3D object detection. In this paper, we propose a method that does not demand any 3D annotation, while being able to predict fully oriented 3D bounding boxes. Our method, called Recursive Cross-View (RCV), utilizes the three-view principle to convert 3D detection into multiple 2D detection tasks, requiring only a subset of 2D labels. We propose a recursive paradigm, in which instance segmentation and 3D bounding box generation by Cross-View are implemented recursively until convergence. Specifically, our proposed method involves the use of a frustum for each 2D bounding box, which is then followed by the recursive paradigm that ultimately generates a fully oriented 3D box, along with its corresponding class and score. Note that, class and score are given by the 2D detector. Estimated on the SUN RGB-D and KITTI datasets, our method outperforms existing image-based approaches. To justify that our method can be quickly used to new tasks, we implement it on two real-world scenarios, namely 3D human detection and 3D hand detection. As a result, two new 3D annotated datasets are obtained, which means that RCV can be viewed as a (semi-) automatic 3D annotator. Furthermore, we deploy RCV on a depth sensor, which achieves detection at 7 fps on a live RGB-D stream. RCV is the first 3D detection method that yields fully oriented 3D boxes without consuming 3D labels.