SM3D: Simultaneous Monocular Mapping and 3D Detection

TL;DR

SM3D is a novel multi-task deep learning framework that simultaneously performs monocular mapping and 3D detection by integrating depth estimation and pseudo-LiDAR point cloud generation, improving accuracy and speed.

Contribution

It introduces an end-to-end trainable system that bridges mapping and 3D detection using pseudo-LiDAR from monocular images, a first in this context.

Findings

Outperforms state-of-the-art in accuracy by 10-13%.

More than twice as fast as stereo 3D detectors.

18.3% faster than separate modules.

Abstract

Mapping and 3D detection are two major issues in vision-based robotics, and self-driving. While previous works only focus on each task separately, we present an innovative and efficient multi-task deep learning framework (SM3D) for Simultaneous Mapping and 3D Detection by bridging the gap with robust depth estimation and "Pseudo-LiDAR" point cloud for the first time. The Mapping module takes consecutive monocular frames to generate depth and pose estimation. In 3D Detection module, the depth estimation is projected into 3D space to generate "Pseudo-LiDAR" point cloud, where LiDAR-based 3D detector can be leveraged on point cloud for vehicular 3D detection and localization. By end-to-end training of both modules, the proposed mapping and 3D detection method outperforms the state-of-the-art baseline by 10.0% and 13.2% in accuracy, respectively. While achieving better accuracy, our monocular multi-task SM3D is more than 2 times faster than pure stereo 3D detector, and 18.3% faster than using two modules separately.