Deeply Supervised Depth Map Super-Resolution as Novel View Synthesis

TL;DR

This paper introduces a novel depth map super-resolution framework that models the task as multiple view synthesis problems, employs deep supervision for large up-sampling factors, and uses multi-scale fusion, outperforming existing methods.

Contribution

It presents a new approach that treats super-resolution as view synthesis, with a deeply supervised network and multi-scale fusion, enabling high-quality results at large up-sampling factors.

Findings

Outperforms state-of-the-art methods on benchmark datasets.

Effectively handles large up-sampling factors like 8x and 16x.

Does not require color images, only low-resolution depth maps.

Abstract

Deep convolutional neural network (DCNN) has been successfully applied to depth map super-resolution and outperforms existing methods by a wide margin. However, there still exist two major issues with these DCNN based depth map super-resolution methods that hinder the performance: i) The low-resolution depth maps either need to be up-sampled before feeding into the network or substantial deconvolution has to be used; and ii) The supervision (high-resolution depth maps) is only applied at the end of the network, thus it is difficult to handle large up-sampling factors, such as $\times 8, \times 16$. In this paper, we propose a new framework to tackle the above problems. First, we propose to represent the task of depth map super-resolution as a series of novel view synthesis sub-tasks. The novel view synthesis sub-task aims at generating (synthesizing) a depth map from different camera pose, which could be learned in parallel. Second, to handle large up-sampling factors, we present a deeply supervised network structure to enforce strong supervision in each stage of the network. Third, a multi-scale fusion strategy is proposed to effectively exploit the feature maps at different scales and handle the blocking effect. In this way, our proposed framework could deal with challenging depth map super-resolution efficiently under large up-sampling factors (e.g. $\times 8, \times 16$). Our method only uses the low-resolution depth map as input, and the support of color image is not needed, which greatly reduces the restriction of our method. Extensive experiments on various benchmarking datasets demonstrate the superiority of our method over current state-of-the-art depth map super-resolution methods.