S2ML: Spatio-Spectral Mutual Learning for Depth Completion

TL;DR

This paper introduces S2ML, a novel depth completion framework that leverages both spatial and spectral domain features, including amplitude and phase spectra, to improve the accuracy of depth images from RGB-D sensors.

Contribution

The paper proposes a spatio-spectral mutual learning approach that explicitly models and fuses spatial and frequency domain features, considering physical properties of raw depth images.

Findings

Outperforms state-of-the-art CFormer by over 0.8 dB on NYU-Depth V2 and SUN RGB-D datasets.

Effectively utilizes spectral fusion and mutual learning for more accurate depth completion.

Demonstrates significant improvements in depth image quality through extensive experiments.

Abstract

The raw depth images captured by RGB-D cameras using Time-of-Flight (TOF) or structured light often suffer from incomplete depth values due to weak reflections, boundary shadows, and artifacts, which limit their applications in downstream vision tasks. Existing methods address this problem through depth completion in the image domain, but they overlook the physical characteristics of raw depth images. It has been observed that the presence of invalid depth areas alters the frequency distribution pattern. In this work, we propose a Spatio-Spectral Mutual Learning framework (S2ML) to harmonize the advantages of both spatial and frequency domains for depth completion. Specifically, we consider the distinct properties of amplitude and phase spectra and devise a dedicated spectral fusion module. Meanwhile, the local and global correlations between spatial-domain and frequency-domain features are calculated in a unified embedding space. The gradual mutual representation and refinement encourage the network to fully explore complementary physical characteristics and priors for more accurate depth completion. Extensive experiments demonstrate the effectiveness of our proposed S2ML method, outperforming the state-of-the-art method CFormer by 0.828 dB and 0.834 dB on the NYU-Depth V2 and SUN RGB-D datasets, respectively.