High-resolution Depth Maps Imaging via Attention-based Hierarchical Multi-modal Fusion

TL;DR

This paper introduces an attention-based hierarchical multi-modal fusion network that significantly improves guided depth map super-resolution by effectively combining low-resolution depth and high-resolution RGB guidance, outperforming existing methods.

Contribution

The paper proposes a novel AHMF network with MMAF and BHFC modules for better feature extraction and fusion in guided DSR, addressing structure consistency and propagation challenges.

Findings

Outperforms state-of-the-art in accuracy

Faster processing speed

More memory-efficient

Abstract

Depth map records distance between the viewpoint and objects in the scene, which plays a critical role in many real-world applications. However, depth map captured by consumer-grade RGB-D cameras suffers from low spatial resolution. Guided depth map super-resolution (DSR) is a popular approach to address this problem, which attempts to restore a high-resolution (HR) depth map from the input low-resolution (LR) depth and its coupled HR RGB image that serves as the guidance. The most challenging problems for guided DSR are how to correctly select consistent structures and propagate them, and properly handle inconsistent ones. In this paper, we propose a novel attention-based hierarchical multi-modal fusion (AHMF) network for guided DSR. Specifically, to effectively extract and combine relevant information from LR depth and HR guidance, we propose a multi-modal attention based fusion (MMAF) strategy for hierarchical convolutional layers, including a feature enhance block to select valuable features and a feature recalibration block to unify the similarity metrics of modalities with different appearance characteristics. Furthermore, we propose a bi-directional hierarchical feature collaboration (BHFC) module to fully leverage low-level spatial information and high-level structure information among multi-scale features. Experimental results show that our approach outperforms state-of-the-art methods in terms of reconstruction accuracy, running speed and memory efficiency.