Cost Volume Pyramid Based Depth Inference for Multi-View Stereo

TL;DR

This paper introduces a multi-view stereo depth inference method using a cost volume pyramid that refines depth maps iteratively, resulting in faster processing and comparable accuracy to state-of-the-art techniques.

Contribution

The novel approach of building a cost volume pyramid for efficient, high-resolution depth inference improves speed and compactness over previous methods like Point-MVSNet.

Findings

Achieves 6x faster inference speed than existing methods.

Maintains similar reconstruction accuracy to state-of-the-art models.

Provides detailed analysis of depth sampling and image resolution relationship.

Abstract

We propose a cost volume-based neural network for depth inference from multi-view images. We demonstrate that building a cost volume pyramid in a coarse-to-fine manner instead of constructing a cost volume at a fixed resolution leads to a compact, lightweight network and allows us inferring high resolution depth maps to achieve better reconstruction results. To this end, we first build a cost volume based on uniform sampling of fronto-parallel planes across the entire depth range at the coarsest resolution of an image. Then, given current depth estimate, we construct new cost volumes iteratively on the pixelwise depth residual to perform depth map refinement. While sharing similar insight with Point-MVSNet as predicting and refining depth iteratively, we show that working on cost volume pyramid can lead to a more compact, yet efficient network structure compared with the Point-MVSNet on 3D points. We further provide detailed analyses of the relation between (residual) depth sampling and image resolution, which serves as a principle for building compact cost volume pyramid. Experimental results on benchmark datasets show that our model can perform 6x faster and has similar performance as state-of-the-art methods. Code is available at https://github.com/JiayuYANG/CVP-MVSNet