Image-Guided Depth Sampling and Reconstruction

TL;DR

This paper introduces an adaptive, image-guided depth sampling and reconstruction method that significantly reduces sampling rates while maintaining high reconstruction quality, leveraging super-pixel based algorithms and a novel depth model.

Contribution

It proposes a new piece-wise linear depth model, a super-pixel based sampling and reconstruction algorithm, and demonstrates state-of-the-art results with reduced sampling rates in depth imaging.

Findings

Optimal depth approximation uses 20-60 linear structures

Sampling rate can be reduced to 1/1200 of image pixels

Reconstruction improves over grid and random sampling methods

Abstract

Depth acquisition, based on active illumination, is essential for autonomous and robotic navigation. LiDARs (Light Detection And Ranging) with mechanical, fixed, sampling templates are commonly used in today's autonomous vehicles. An emerging technology, based on solid-state depth sensors, with no mechanical parts, allows fast, adaptive, programmable scans. In this paper, we investigate the topic of adaptive, image-driven, sampling and reconstruction strategies. First, we formulate a piece-wise linear depth model with several tolerance parameters and estimate its validity for indoor and outdoor scenes. Our model and experiments predict that, in the optimal case, about 20-60 piece-wise linear structures can approximate well a depth map. This translates to a depth-to-image sampling ratio of about 1/1200. We propose a simple, generic, sampling and reconstruction algorithm, based on super-pixels. We reach a sampling rate which is still far from the optimal case. However, our sampling improves grid and random sampling, consistently, for a wide variety of reconstruction methods. Moreover, our proposed reconstruction achieves state-of-the-art results, compared to image-guided depth completion algorithms, reducing the required sampling rate by a factor of 3-4. A single-pixel depth camera built in our lab illustrates the concept.