Single-View and Multi-View Depth Fusion

TL;DR

This paper proposes a fusion method combining single-view CNN-based and multi-view depth estimation techniques to improve dense 3D mapping accuracy, especially in texture-less and low-parallax regions.

Contribution

It introduces a novel fusion approach that addresses deformation and point selection challenges, outperforming individual methods on standard datasets.

Findings

Outperforms individual single-view and multi-view depth methods on NYUv2 and TUM datasets.

Effectively captures local structures in texture-less areas.

Enhances 3D mapping accuracy through combined depth estimation.

Abstract

Dense and accurate 3D mapping from a monocular sequence is a key technology for several applications and still an open research area. This paper leverages recent results on single-view CNN-based depth estimation and fuses them with multi-view depth estimation. Both approaches present complementary strengths. Multi-view depth is highly accurate but only in high-texture areas and high-parallax cases. Single-view depth captures the local structure of mid-level regions, including texture-less areas, but the estimated depth lacks global coherence. The single and multi-view fusion we propose is challenging in several aspects. First, both depths are related by a deformation that depends on the image content. Second, the selection of multi-view points of high accuracy might be difficult for low-parallax configurations. We present contributions for both problems. Our results in the public datasets of NYUv2 and TUM shows that our algorithm outperforms the individual single and multi-view approaches. A video showing the key aspects of mapping in our Single and Multi-view depth proposal is available at https://youtu.be/ipc5HukTb4k