DepthNet: Real-Time LiDAR Point Cloud Depth Completion for Autonomous Vehicles

TL;DR

DepthNet introduces a lightweight, real-time LiDAR depth completion network for autonomous vehicles, significantly reducing parameters while maintaining performance, and implementing FPGA-based processing for embedded systems.

Contribution

A novel, highly efficient neural network architecture for LiDAR depth completion that enables real-time processing on embedded platforms with minimal parameter count.

Findings

Achieves 96.2% reduction in parameters with comparable accuracy to state-of-the-art.

Further reduces parameters by 7.3 times using depthwise separable convolutions.

Enables real-time depth completion on FPGA at 11.1 fps.

Abstract

Autonomous vehicles rely heavily on sensors such as camera and LiDAR, which provide real-time information about their surroundings for the tasks of perception, planning and control. Typically a LiDAR can only provide sparse point cloud owing to a limited number of scanning lines. By employing depth completion, a dense depth map can be generated by assigning each camera pixel a corresponding depth value. However, the existing depth completion convolutional neural networks are very complex that requires high-end GPUs for processing, and thus they are not applicable to real-time autonomous driving. In this paper, a light-weight network is proposed for the task of LiDAR point cloud depth completion. With an astonishing 96.2% reduction in the number of parameters, it still achieves comparable performance (9.3% better in MAE but 3.9% worse in RMSE) to the state-of-the-art network. For real-time embedded platforms, depthwise separable technique is applied to both convolution and deconvolution operations and the number of parameters decreases further by a factor of 7.3, with only a small percentage increase in RMSE and MAE performance. Moreover, a system-on-chip architecture for depth completion is developed on a PYNQ-based FPGA platform that achieves real-time processing for HDL-64E LiDAR at the speed 11.1 frame per second.