Boosting Multi-view Stereo with Late Cost Aggregation

TL;DR

This paper introduces a late cost aggregation method for multi-view stereo that preserves pairwise matching costs throughout the network, leading to more accurate depth estimation without significant computational increase.

Contribution

The paper proposes a novel late aggregation scheme for MVS that maintains pairwise costs during processing, improving accuracy over early aggregation methods.

Findings

Significant accuracy improvement over baseline cascade-based approach

Achieves state-of-the-art results with minimal additional computation

Effectively handles view order dependence and flexible testing views

Abstract

Pairwise matching cost aggregation is a crucial step for modern learning-based Multi-view Stereo (MVS). Prior works adopt an early aggregation scheme, which adds up pairwise costs into an intermediate cost. However, we analyze that this process can degrade informative pairwise matchings, thereby blocking the depth network from fully utilizing the original geometric matching cues. To address this challenge, we present a late aggregation approach that allows for aggregating pairwise costs throughout the network feed-forward process, achieving accurate estimations with only minor changes of the plain CasMVSNet. Instead of building an intermediate cost by weighted sum, late aggregation preserves all pairwise costs along a distinct view channel. This enables the succeeding depth network to fully utilize the crucial geometric cues without loss of cost fidelity. Grounded in the new aggregation scheme, we propose further techniques addressing view order dependence inside the preserved cost, handling flexible testing views, and improving the depth filtering process. Despite its technical simplicity, our method improves significantly upon the baseline cascade-based approach, achieving comparable results with state-of-the-art methods with favorable computation overhead.