# Perceptual deep depth super-resolution

## TL;DR

This paper presents a deep learning-based method for super-resolving depth maps by optimizing visual appearance in 3D renderings, leading to improved 3D shape quality for applications like virtual reality.

## Contribution

It introduces a novel perceptual loss based on 3D surface renderings for depth super-resolution, enhancing shape quality over existing methods.

## Key findings

- Significant improvement in 3D shape quality using the proposed perceptual loss.
- Effective use of deep prior and CNN-based models for depth upsampling.
- Outperforms existing optimization and learning-based techniques in perceptual metrics.

## Abstract

RGBD images, combining high-resolution color and lower-resolution depth from various types of depth sensors, are increasingly common. One can significantly improve the resolution of depth maps by taking advantage of color information; deep learning methods make combining color and depth information particularly easy. However, fusing these two sources of data may lead to a variety of artifacts. If depth maps are used to reconstruct 3D shapes, e.g., for virtual reality applications, the visual quality of upsampled images is particularly important. The main idea of our approach is to measure the quality of depth map upsampling using renderings of resulting 3D surfaces. We demonstrate that a simple visual appearance-based loss, when used with either a trained CNN or simply a deep prior, yields significantly improved 3D shapes, as measured by a number of existing perceptual metrics. We compare this approach with a number of existing optimization and learning-based techniques.