Non-local Recurrent Regularization Networks for Multi-view Stereo

TL;DR

This paper introduces NR2-Net, a novel non-local recurrent regularization network for multi-view stereo that captures global scene context along the depth dimension, improving 3D reconstruction accuracy.

Contribution

The paper proposes a non-local depth attention module and a gated recurrent approach to model long-range dependencies in depth, advancing multi-view stereo regularization techniques.

Findings

Achieves state-of-the-art results on DTU dataset

Outperforms existing methods on Tanks and Temples dataset

Improves robustness with dynamic depth map fusion

Abstract

In deep multi-view stereo networks, cost regularization is crucial to achieve accurate depth estimation. Since 3D cost volume filtering is usually memory-consuming, recurrent 2D cost map regularization has recently become popular and has shown great potential in reconstructing 3D models of different scales. However, existing recurrent methods only model the local dependencies in the depth domain, which greatly limits the capability of capturing the global scene context along the depth dimension. To tackle this limitation, we propose a novel non-local recurrent regularization network for multi-view stereo, named NR2-Net. Specifically, we design a depth attention module to capture non-local depth interactions within a sliding depth block. Then, the global scene context between different blocks is modeled in a gated recurrent manner. This way, the long-range dependencies along the depth dimension are captured to facilitate the cost regularization. Moreover, we design a dynamic depth map fusion strategy to improve the algorithm robustness. Our method achieves state-of-the-art reconstruction results on both DTU and Tanks and Temples datasets.