Lifting 2D Object Locations to 3D by Discounting LiDAR Outliers across Objects and Views

TL;DR

This paper introduces a novel system that converts 2D object masks and LiDAR data into accurate 3D bounding boxes by jointly leveraging multiple frames, explicitly modeling outliers, and optimizing rotation predictions.

Contribution

The method's key innovations include joint multi-object information sharing, explicit outlier modeling, and temporal consistency enforcement, improving 3D object localization from partial LiDAR data.

Findings

Significantly outperforms previous methods

Achieves high accuracy with simpler pipeline

Effectively discounts LiDAR outliers

Abstract

We present a system for automatic converting of 2D mask object predictions and raw LiDAR point clouds into full 3D bounding boxes of objects. Because the LiDAR point clouds are partial, directly fitting bounding boxes to the point clouds is meaningless. Instead, we suggest that obtaining good results requires sharing information between \emph{all} objects in the dataset jointly, over multiple frames. We then make three improvements to the baseline. First, we address ambiguities in predicting the object rotations via direct optimization in this space while still backpropagating rotation prediction through the model. Second, we explicitly model outliers and task the network with learning their typical patterns, thus better discounting them. Third, we enforce temporal consistency when video data is available. With these contributions, our method significantly outperforms previous work despite the fact that those methods use significantly more complex pipelines, 3D models and additional human-annotated external sources of prior information.