Scene Prior Filtering for Depth Super-Resolution

TL;DR

This paper introduces SPFNet, a novel depth super-resolution method that leverages multi-modal scene priors and mutual guided filtering to effectively reduce texture interference and improve edge accuracy, achieving state-of-the-art results.

Contribution

It proposes a Scene Prior Filtering network with all-in-one prior propagation and one-to-one prior embedding to enhance depth super-resolution by better utilizing multi-modal priors.

Findings

Achieves state-of-the-art performance on real-world datasets.

Effectively reduces texture interference in depth maps.

Improves edge accuracy through mutual guided filtering.

Abstract

Multi-modal fusion serves as a cornerstone for successful depth map super-resolution. However, commonly used fusion strategies, such as addition and concatenation, fall short of effectively bridging the modal gap. As a result, guided image filtering methods have been introduced to mitigate this issue. Nevertheless, it is observed that their filter kernels usually encounter significant texture interference and edge inaccuracy. To tackle these two challenges, we introduce a Scene Prior Filtering network, SPFNet, which utilizes the priors' surface normal and semantic map from large-scale models. Specifically, we propose an All-in-one Prior Propagation that computes similarity between multi-modal scene priors, i.e., RGB, normal, semantic, and depth, to reduce the texture interference. Besides, we design a One-to-one Prior Embedding that continuously embeds every single modal prior into depth using Mutual Guided Filtering, further alleviating texture interference while enhancing edge representations. Our SPFNet has been extensively evaluated on both real-world and synthetic datasets, achieving state-of-the-art performance.