ES-MVSNet: Efficient Framework for End-to-end Self-supervised Multi-View Stereo

TL;DR

ES-MVSNet introduces a memory-efficient end-to-end self-supervised multi-view stereo framework that achieves state-of-the-art results without third-party models, reducing memory usage significantly while maintaining high performance.

Contribution

The paper presents a novel, memory-efficient architecture and a new view selection policy for end-to-end self-supervised MVS, eliminating the need for external models.

Findings

Reduces memory consumption by 43% without performance loss.

Achieves state-of-the-art results among E2E self-supervised MVS methods.

Performs competitively with supervised and multi-stage self-supervised approaches.

Abstract

Compared to the multi-stage self-supervised multi-view stereo (MVS) method, the end-to-end (E2E) approach has received more attention due to its concise and efficient training pipeline. Recent E2E self-supervised MVS approaches have integrated third-party models (such as optical flow models, semantic segmentation models, NeRF models, etc.) to provide additional consistency constraints, which grows GPU memory consumption and complicates the model's structure and training pipeline. In this work, we propose an efficient framework for end-to-end self-supervised MVS, dubbed ES-MVSNet. To alleviate the high memory consumption of current E2E self-supervised MVS frameworks, we present a memory-efficient architecture that reduces memory usage by 43% without compromising model performance. Furthermore, with the novel design of asymmetric view selection policy and region-aware depth consistency, we achieve state-of-the-art performance among E2E self-supervised MVS methods, without relying on third-party models for additional consistency signals. Extensive experiments on DTU and Tanks&Temples benchmarks demonstrate that the proposed ES-MVSNet approach achieves state-of-the-art performance among E2E self-supervised MVS methods and competitive performance to many supervised and multi-stage self-supervised methods.