End-to-End Multi-View Structure-from-Motion with Hypercorrelation Volumes

TL;DR

This paper introduces an end-to-end deep learning approach for multi-view structure-from-motion that uses 4D correlation volumes to improve feature matching and 3D reconstruction accuracy, outperforming existing methods.

Contribution

It extends a two-view SfM method to the multi-view case with 4D correlation volumes, achieving more accurate 3D reconstructions in complex datasets.

Findings

Outperforms state-of-the-art multi-view 3D reconstruction methods

Demonstrates superior accuracy on the DTU benchmark dataset

Validates effectiveness of 4D correlation volumes in SfM

Abstract

Image-based 3D reconstruction is one of the most important tasks in Computer Vision with many solutions proposed over the last few decades. The objective is to extract metric information i.e. the geometry of scene objects directly from images. These can then be used in a wide range of applications such as film, games, virtual reality, etc. Recently, deep learning techniques have been proposed to tackle this problem. They rely on training on vast amounts of data to learn to associate features between images through deep convolutional neural networks and have been shown to outperform traditional procedural techniques. In this paper, we improve on the state-of-the-art two-view structure-from-motion(SfM) approach of [11] by incorporating 4D correlation volume for more accurate feature matching and reconstruction. Furthermore, we extend it to the general multi-view case and evaluate it on the complex benchmark dataset DTU [4]. Quantitative evaluations and comparisons with state-of-the-art multi-view 3D reconstruction methods demonstrate its superiority in terms of the accuracy of reconstructions.