Joint stereo 3D object detection and implicit surface reconstruction

TL;DR

This paper introduces a new framework for 3D object detection and shape reconstruction from stereo images, improving orientation estimation and surface modeling.

Contribution

The novel contribution is a progressive method using Intermediate Geometrical Representations for orientation and implicit shape estimation from stereo RGB images.

Findings

The proposed IGRs improve orientation estimation accuracy in SO(3) compared to previous methods.

S-3D-RCNN achieves superior 3D scene understanding performance on benchmark datasets.

New metrics were developed for evaluating implicit shape estimation on the KITTI benchmark.

Abstract

We present a new learning-based framework S-3D-RCNN that can recover accurate object orientation in SO(3) and simultaneously predict implicit rigid shapes from stereo RGB images. For orientation estimation, in contrast to previous studies that map local appearance to observation angles, we propose a progressive approach by extracting meaningful Intermediate Geometrical Representations (IGRs). This approach features a deep model that transforms perceived intensities from one or two views to object part coordinates to achieve direct egocentric object orientation estimation in the camera coordinate system. To further achieve finer description inside 3D bounding boxes, we investigate the implicit shape estimation problem from stereo images. We model visible object surfaces by designing a point-based representation, augmenting IGRs to explicitly address the unseen surface hallucination problem. Extensive experiments validate the effectiveness of the proposed IGRs, and S-3D-RCNN achieves superior 3D scene understanding performance. We also designed new metrics on the KITTI benchmark for our evaluation of implicit shape estimation.

Figures