Self-supervised Sparse-to-Dense: Self-supervised Depth Completion from LiDAR and Monocular Camera

TL;DR

This paper introduces a self-supervised deep learning approach for depth completion that effectively combines LiDAR and monocular camera data, overcoming challenges of irregular sparse inputs and lack of dense ground truth, achieving state-of-the-art results.

Contribution

It presents a novel self-supervised training framework and a deep regression model that directly maps sparse depth and images to dense depth without dense labels.

Findings

Achieves state-of-the-art accuracy on KITTI benchmark.

Outperforms existing semi-supervised methods in self-supervised setting.

Effective handling of irregular sparse inputs and multiple modalities.

Abstract

Depth completion, the technique of estimating a dense depth image from sparse depth measurements, has a variety of applications in robotics and autonomous driving. However, depth completion faces 3 main challenges: the irregularly spaced pattern in the sparse depth input, the difficulty in handling multiple sensor modalities (when color images are available), as well as the lack of dense, pixel-level ground truth depth labels. In this work, we address all these challenges. Specifically, we develop a deep regression model to learn a direct mapping from sparse depth (and color images) to dense depth. We also propose a self-supervised training framework that requires only sequences of color and sparse depth images, without the need for dense depth labels. Our experiments demonstrate that our network, when trained with semi-dense annotations, attains state-of-the- art accuracy and is the winning approach on the KITTI depth completion benchmark at the time of submission. Furthermore, the self-supervised framework outperforms a number of existing solutions trained with semi- dense annotations.