Lifting Object Detection Datasets into 3D

TL;DR

This paper introduces a semi-automatic method to generate dense 3D reconstructions of objects in detection datasets using class labels, segmentations, and keypoints, improving 3D data availability for recognition tasks.

Contribution

It presents a novel approach that bypasses costly 3D scanning by reconstructing 3D shapes from 2D images using shape-from-motion and visual hulls, enhancing dataset richness.

Findings

Successfully reconstructs 3D shapes on PASCAL VOC dataset

Accurately estimates camera viewpoints from 2D data

Produces convincing 3D models from limited annotations

Abstract

While data has certainly taken the center stage in computer vision in recent years, it can still be difficult to obtain in certain scenarios. In particular, acquiring ground truth 3D shapes of objects pictured in 2D images remains a challenging feat and this has hampered progress in recognition-based object reconstruction from a single image. Here we propose to bypass previous solutions such as 3D scanning or manual design, that scale poorly, and instead populate object category detection datasets semi-automatically with dense, per-object 3D reconstructions, bootstrapped from:(i) class labels, (ii) ground truth figure-ground segmentations and (iii) a small set of keypoint annotations. Our proposed algorithm first estimates camera viewpoint using rigid structure-from-motion and then reconstructs object shapes by optimizing over visual hull proposals guided by loose within-class shape similarity assumptions. The visual hull sampling process attempts to intersect an object's projection cone with the cones of minimal subsets of other similar objects among those pictured from certain vantage points. We show that our method is able to produce convincing per-object 3D reconstructions and to accurately estimate cameras viewpoints on one of the most challenging existing object-category detection datasets, PASCAL VOC. We hope that our results will re-stimulate interest on joint object recognition and 3D reconstruction from a single image.