Prism: Semi-Supervised Multi-View Stereo with Monocular Structure Priors

TL;DR

Prism introduces a semi-supervised learning framework for multi-view stereo that combines synthetic and real data, leveraging monocular depth priors and novel loss functions to improve 3D reconstruction accuracy.

Contribution

It presents a new semi-supervised approach that effectively integrates synthetic and real data for MVS training using monocular priors and perceptual losses.

Findings

Significant improvements over existing unsupervised MVS methods.

Effective transfer of synthetic structural priors to real-world data.

Demonstrated potential for training MVS with unlabeled smartphone videos.

Abstract

The promise of unsupervised multi-view-stereo (MVS) is to leverage large unlabeled datasets, yet current methods underperform when training on difficult data, such as handheld smartphone videos of indoor scenes. Meanwhile, high-quality synthetic datasets are available but MVS networks trained on these datasets fail to generalize to real-world examples. To bridge this gap, we propose a semi-supervised learning framework that allows us to train on real and rendered images jointly, capturing structural priors from synthetic data while ensuring parity with the real-world domain. Central to our framework is a novel set of losses that leverages powerful existing monocular relative-depth estimators trained on the synthetic dataset, transferring the rich structure of this relative depth to the MVS predictions on unlabeled data. Inspired by perceptual image metrics, we compare the MVS and monocular predictions via a deep feature loss and a multi-scale statistical loss. Our full framework, which we call Prism, achieves large quantitative and qualitative improvements over current unsupervised and synthetic-supervised MVS networks. This is a best-case-scenario result, opening the door to using both unlabeled smartphone videos and photorealistic synthetic datasets for training MVS networks.