Sparse LiDAR and Stereo Fusion (SLS-Fusion) for Depth Estimationand 3D Object Detection

TL;DR

This paper introduces SLS-Fusion, a neural network-based method that fuses low-cost 4-beam LiDAR with stereo camera data to improve depth estimation and 3D object detection in autonomous vehicles.

Contribution

It is the first to fuse LiDAR and stereo data in a deep neural network for 3D detection, achieving state-of-the-art results with low-cost sensors.

Findings

Significant improvement in depth estimation over baseline methods.

Achieved new state-of-the-art 3D detection performance with low-cost sensors.

Validated on the KITTI benchmark.

Abstract

The ability to accurately detect and localize objects is recognized as being the most important for the perception of self-driving cars. From 2D to 3D object detection, the most difficult is to determine the distance from the ego-vehicle to objects. Expensive technology like LiDAR can provide a precise and accurate depth information, so most studies have tended to focus on this sensor showing a performance gap between LiDAR-based methods and camera-based methods. Although many authors have investigated how to fuse LiDAR with RGB cameras, as far as we know there are no studies to fuse LiDAR and stereo in a deep neural network for the 3D object detection task. This paper presents SLS-Fusion, a new approach to fuse data from 4-beam LiDAR and a stereo camera via a neural network for depth estimation to achieve better dense depth maps and thereby improves 3D object detection performance. Since 4-beam LiDAR is cheaper than the well-known 64-beam LiDAR, this approach is also classified as a low-cost sensors-based method. Through evaluation on the KITTI benchmark, it is shown that the proposed method significantly improves depth estimation performance compared to a baseline method. Also, when applying it to 3D object detection, a new state of the art on low-cost sensor based method is achieved.