Deep Stereo using Adaptive Thin Volume Representation with Uncertainty Awareness

TL;DR

This paper introduces UCS-Net, a multi-view stereo method that uses adaptive thin volumes with uncertainty estimates to improve 3D scene reconstruction accuracy and resolution.

Contribution

It proposes a novel adaptive thin volume representation guided by variance-based uncertainty, enabling efficient high-resolution depth estimation in multi-view stereo.

Findings

Outperforms state-of-the-art methods on challenging datasets.

Achieves higher depth accuracy and scene completeness.

Efficiently refines depth with fewer planes using adaptive partitioning.

Abstract

We present Uncertainty-aware Cascaded Stereo Network (UCS-Net) for 3D reconstruction from multiple RGB images. Multi-view stereo (MVS) aims to reconstruct fine-grained scene geometry from multi-view images. Previous learning-based MVS methods estimate per-view depth using plane sweep volumes with a fixed depth hypothesis at each plane; this generally requires densely sampled planes for desired accuracy, and it is very hard to achieve high-resolution depth. In contrast, we propose adaptive thin volumes (ATVs); in an ATV, the depth hypothesis of each plane is spatially varying, which adapts to the uncertainties of previous per-pixel depth predictions. Our UCS-Net has three stages: the first stage processes a small standard plane sweep volume to predict low-resolution depth; two ATVs are then used in the following stages to refine the depth with higher resolution and higher accuracy. Our ATV consists of only a small number of planes; yet, it efficiently partitions local depth ranges within learned small intervals. In particular, we propose to use variance-based uncertainty estimates to adaptively construct ATVs; this differentiable process introduces reasonable and fine-grained spatial partitioning. Our multi-stage framework progressively subdivides the vast scene space with increasing depth resolution and precision, which enables scene reconstruction with high completeness and accuracy in a coarse-to-fine fashion. We demonstrate that our method achieves superior performance compared with state-of-the-art benchmarks on various challenging datasets.