Photometric Depth Super-Resolution

TL;DR

This paper presents novel photometric depth super-resolution methods that enhance low-resolution depth maps using shape-from-shading and photometric stereo, employing variational, neural, and multi-shot strategies for improved accuracy.

Contribution

It introduces a variational approach for depth upsampling, relaxes reflectance assumptions with neural networks, and discusses a multi-shot method requiring no prior reflectance knowledge.

Findings

Effective depth upsampling on synthetic and real data

Neural network-based reflectance estimation improves results

Multi-shot approach achieves high accuracy without prior reflectance

Abstract

This study explores the use of photometric techniques (shape-from-shading and uncalibrated photometric stereo) for upsampling the low-resolution depth map from an RGB-D sensor to the higher resolution of the companion RGB image. A single-shot variational approach is first put forward, which is effective as long as the target's reflectance is piecewise-constant. It is then shown that this dependency upon a specific reflectance model can be relaxed by focusing on a specific class of objects (e.g., faces), and delegate reflectance estimation to a deep neural network. A multi-shot strategy based on randomly varying lighting conditions is eventually discussed. It requires no training or prior on the reflectance, yet this comes at the price of a dedicated acquisition setup. Both quantitative and qualitative evaluations illustrate the effectiveness of the proposed methods on synthetic and real-world scenarios.