MASt3R-SfM: a Fully-Integrated Solution for Unconstrained Structure-from-Motion

TL;DR

This paper introduces MASt3R-SfM, a fully integrated, scalable, and unconstrained structure-from-motion pipeline leveraging foundation models for robust local 3D reconstructions, accurate global alignment, and efficient image retrieval, outperforming existing methods.

Contribution

It presents a novel SfM approach that integrates foundation models for local reconstruction, global alignment, and image retrieval, addressing core challenges of traditional pipelines.

Findings

Outperforms existing SfM methods on multiple benchmarks.

Handles any collection of images, ordered or not.

Provides steady performance across diverse settings.

Abstract

Structure-from-Motion (SfM), a task aiming at jointly recovering camera poses and 3D geometry of a scene given a set of images, remains a hard problem with still many open challenges despite decades of significant progress. The traditional solution for SfM consists of a complex pipeline of minimal solvers which tends to propagate errors and fails when images do not sufficiently overlap, have too little motion, etc. Recent methods have attempted to revisit this paradigm, but we empirically show that they fall short of fixing these core issues. In this paper, we propose instead to build upon a recently released foundation model for 3D vision that can robustly produce local 3D reconstructions and accurate matches. We introduce a low-memory approach to accurately align these local reconstructions in a global coordinate system. We further show that such foundation models can serve as efficient image retrievers without any overhead, reducing the overall complexity from quadratic to linear. Overall, our novel SfM pipeline is simple, scalable, fast and truly unconstrained, i.e. it can handle any collection of images, ordered or not. Extensive experiments on multiple benchmarks show that our method provides steady performance across diverse settings, especially outperforming existing methods in small- and medium-scale settings.