Multi view stereo with semantic priors

TL;DR

This paper enhances multi-view stereo 3D reconstruction by integrating semantic priors into the fusion process, improving accuracy and automation through semantic constraints and segmentation.

Contribution

It introduces a method to incorporate semantic priors into the depth map fusion step of multi-view stereo, improving scene reconstruction accuracy and semantic consistency.

Findings

Semantic priors improve 3D reconstruction accuracy.

Semantic constraints reduce mismatches between views.

Segmentation per label enhances automation.

Abstract

Patch-based stereo is nowadays a commonly used image-based technique for dense 3D reconstruction in large scale multi-view applications. The typical steps of such a pipeline can be summarized in stereo pair selection, depth map computation, depth map refinement and, finally, fusion in order to generate a complete and accurate representation of the scene in 3D. In this study, we aim to support the standard dense 3D reconstruction of scenes as implemented in the open source library OpenMVS by using semantic priors. To this end, during the depth map fusion step, along with the depth consistency check between depth maps of neighbouring views referring to the same part of the 3D scene, we impose extra semantic constraints in order to remove possible errors and selectively obtain segmented point clouds per label, boosting automation towards this direction. I n order to reassure semantic coherence between neighbouring views, additional semantic criterions can be considered, aiming to elim inate mismatches of pixels belonging in different classes.