Estimating Depth from RGB and Sparse Sensing

TL;DR

This paper introduces a deep learning model that converts sparse depth measurements into dense depth maps with high accuracy, working efficiently across indoor and outdoor scenes and outperforming existing methods.

Contribution

The authors develop a novel deep model capable of producing dense depth maps from sparse depth data, achieving state-of-the-art accuracy at real-time speeds and with minimal depth input.

Findings

Surpasses state-of-the-art monocular depth estimation with sparse data.

Achieves less than 1% mean absolute error with 1/256 pixel depth data.

Works effectively on both indoor and outdoor datasets.

Abstract

We present a deep model that can accurately produce dense depth maps given an RGB image with known depth at a very sparse set of pixels. The model works simultaneously for both indoor/outdoor scenes and produces state-of-the-art dense depth maps at nearly real-time speeds on both the NYUv2 and KITTI datasets. We surpass the state-of-the-art for monocular depth estimation even with depth values for only 1 out of every ~10000 image pixels, and we outperform other sparse-to-dense depth methods at all sparsity levels. With depth values for 1/256 of the image pixels, we achieve a mean absolute error of less than 1% of actual depth on indoor scenes, comparable to the performance of consumer-grade depth sensor hardware. Our experiments demonstrate that it would indeed be possible to efficiently transform sparse depth measurements obtained using e.g. lower-power depth sensors or SLAM systems into high-quality dense depth maps.