Unpaired Depth Super-Resolution in the Wild

TL;DR

This paper introduces an unpaired learning approach for depth map super-resolution that effectively handles real-world low-resolution data without requiring paired training samples, outperforming existing unpaired methods.

Contribution

The authors propose a novel unpaired depth super-resolution method using a learnable degradation model and surface normal features, advancing beyond prior paired-data reliance.

Findings

Outperforms existing unpaired methods in depth super-resolution

Performs comparably to supervised paired-data methods

Introduces a new benchmark for unpaired depth SR

Abstract

Depth maps captured with commodity sensors are often of low quality and resolution; these maps need to be enhanced to be used in many applications. State-of-the-art data-driven methods of depth map super-resolution rely on registered pairs of low- and high-resolution depth maps of the same scenes. Acquisition of real-world paired data requires specialized setups. Another alternative, generating low-resolution maps from high-resolution maps by subsampling, adding noise and other artificial degradation methods, does not fully capture the characteristics of real-world low-resolution images. As a consequence, supervised learning methods trained on such artificial paired data may not perform well on real-world low-resolution inputs. We consider an approach to depth super-resolution based on learning from unpaired data. While many techniques for unpaired image-to-image translation have been proposed, most fail to deliver effective hole-filling or reconstruct accurate surfaces using depth maps. We propose an unpaired learning method for depth super-resolution, which is based on a learnable degradation model, enhancement component and surface normal estimates as features to produce more accurate depth maps. We propose a benchmark for unpaired depth SR and demonstrate that our method outperforms existing unpaired methods and performs on par with paired.