Reference-Free Omnidirectional Stereo Matching via Multi-View Consistency Maximization

TL;DR

This paper introduces FreeOmniMVS, a novel reference-free omnidirectional stereo matching framework that maximizes multi-view consistency to improve depth estimation by explicitly modeling pairwise correlations and fusing them with a lightweight attention mechanism.

Contribution

The paper proposes a new reference-free framework with a View-pair Correlation Transformer and adaptive fusion for improved global, visibility-aware omnidirectional stereo matching.

Findings

Outperforms existing methods on benchmark datasets.

Achieves globally consistent and scale-aware depth estimation.

Effectively handles occlusions and partial overlaps.

Abstract

Reliable omnidirectional depth estimation from multi-fisheye stereo matching is pivotal to many applications, such as embodied robotics. Existing approaches either rely on spherical sweeping with heuristic fusion strategies to build the cost columns or perform reference-centric stereo matching based on rectified views. However, these methods fail to explicitly exploit geometric relationships between multiple views, rendering them less capable of capturing the global dependencies, visibility, or scale changes. In this paper, we shift to a new perspective and propose a novel reference-free framework, dubbed FreeOmniMVS, via multi-view consistency maximization. The highlight of FreeOmniMVS is that it can aggregate pair-wise correlations into a robust, visibility-aware, and global consensus. As such, it is tolerant to occlusions, partial overlaps, and varying baselines. Specifically, to achieve global coherence, we introduce a novel View-pair Correlation Transformer (VCT) that explicitly models pairwise correlation volumes across all camera view pairs, allowing us to drop unreliable pairs caused by occlusion or out-of-focus observations. To realize scalable and visibility-aware consensus, we propose a lightweight attention mechanism that adaptively fuses the correlation vectors, eliminating the need for a designated reference view and allowing all cameras to contribute equally to the stereo matching process. Extensive experiments on diverse benchmark datasets demonstrate the superiority of our method for globally consistent, visibility-aware, and scale-aware omnidirectional depth estimation.