Revisiting 360 Depth Estimation with PanoGabor: A New Fusion Perspective

TL;DR

This paper introduces PGFuse, a novel depth estimation framework for 360 images that uses distortion-aware Gabor filters and a fusion module to improve feature extraction and reduce distortions, outperforming existing methods.

Contribution

The work proposes a new Gabor filter-based fusion approach with distortion-aware design and a spherical gradient constraint for improved 360 depth estimation.

Findings

Outperforms state-of-the-art on indoor 360 benchmarks

Enhances depth cues with frequency domain texture analysis

Effectively unifies representations into distortion-free features

Abstract

Depth estimation from a monocular 360 image is important to the perception of the entire 3D environment. However, the inherent distortion and large field of view (FoV) in 360 images pose great challenges for this task. To this end, existing mainstream solutions typically introduce additional perspective-based 360 representations ({e.g., Cubemap) to achieve effective feature extraction. Nevertheless, regardless of the introduced representations, they eventually need to be unified into the equirectangular projection (ERP) format for the subsequent depth estimation, which inevitably reintroduces the troublesome distortions. In this work, we propose an oriented distortion-aware Gabor Fusion framework (PGFuse) to address the above challenges. First, we introduce Gabor filters that analyze texture in the frequency domain, thereby extending the receptive fields and enhancing depth cues. To address the reintroduced distortions, we design a linear latitude-aware distortion representation method to generate customized, distortion-aware Gabor filters (PanoGabor filters). Furthermore, we design a channel-wise and spatial-wise unidirectional fusion module (CS-UFM) that integrates the proposed PanoGabor filters to unify other representations into the ERP format, delivering effective and distortion-free features. Considering the orientation sensitivity of the Gabor transform, we introduce a spherical gradient constraint to stabilize this sensitivity. Experimental results on three popular indoor 360 benchmarks demonstrate the superiority of the proposed PGFuse to existing state-of-the-art solutions. Code and models will be available at https://github.com/zhijieshen-bjtu/PGFuse