Unsupervised Depth Completion with Calibrated Backprojection Layers

TL;DR

This paper introduces KBNet, a deep neural network that infers dense depth maps from images and sparse point clouds using calibrated backprojection layers, trained without supervision, and adaptable to different camera calibrations.

Contribution

The paper presents a novel unsupervised depth completion network utilizing calibrated backprojection layers that handle varying camera calibrations and improve accuracy significantly.

Findings

Outperforms state-of-the-art by 30.5% indoors and 8.8% outdoors with same camera calibration.

Achieves 62% improvement when test camera calibration differs from training.

Operates effectively without supervision by minimizing photometric reprojection error.

Abstract

We propose a deep neural network architecture to infer dense depth from an image and a sparse point cloud. It is trained using a video stream and corresponding synchronized sparse point cloud, as obtained from a LIDAR or other range sensor, along with the intrinsic calibration parameters of the camera. At inference time, the calibration of the camera, which can be different than the one used for training, is fed as an input to the network along with the sparse point cloud and a single image. A Calibrated Backprojection Layer backprojects each pixel in the image to three-dimensional space using the calibration matrix and a depth feature descriptor. The resulting 3D positional encoding is concatenated with the image descriptor and the previous layer output to yield the input to the next layer of the encoder. A decoder, exploiting skip-connections, produces a dense depth map. The resulting Calibrated Backprojection Network, or KBNet, is trained without supervision by minimizing the photometric reprojection error. KBNet imputes missing depth value based on the training set, rather than on generic regularization. We test KBNet on public depth completion benchmarks, where it outperforms the state of the art by 30.5% indoor and 8.8% outdoor when the same camera is used for training and testing. When the test camera is different, the improvement reaches 62%. Code available at: https://github.com/alexklwong/calibrated-backprojection-network.