Structure Flow-Guided Network for Real Depth Super-Resolution

TL;DR

This paper introduces a novel structure flow-guided network for real-world depth super-resolution, effectively addressing structural distortions and noise in low-resolution depth maps by leveraging cross-modality flow guidance.

Contribution

It proposes a new framework with a flow-guided upsampling network and an edge attention network to improve depth super-resolution accuracy in real-world scenarios.

Findings

Outperforms state-of-the-art methods on real and synthetic datasets.

Effectively reduces structural distortion and edge noise.

Enhances depth map accuracy and detail preservation.

Abstract

Real depth super-resolution (DSR), unlike synthetic settings, is a challenging task due to the structural distortion and the edge noise caused by the natural degradation in real-world low-resolution (LR) depth maps. These defeats result in significant structure inconsistency between the depth map and the RGB guidance, which potentially confuses the RGB-structure guidance and thereby degrades the DSR quality. In this paper, we propose a novel structure flow-guided DSR framework, where a cross-modality flow map is learned to guide the RGB-structure information transferring for precise depth upsampling. Specifically, our framework consists of a cross-modality flow-guided upsampling network (CFUNet) and a flow-enhanced pyramid edge attention network (PEANet). CFUNet contains a trilateral self-attention module combining both the geometric and semantic correlations for reliable cross-modality flow learning. Then, the learned flow maps are combined with the grid-sampling mechanism for coarse high-resolution (HR) depth prediction. PEANet targets at integrating the learned flow map as the edge attention into a pyramid network to hierarchically learn the edge-focused guidance feature for depth edge refinement. Extensive experiments on real and synthetic DSR datasets verify that our approach achieves excellent performance compared to state-of-the-art methods.