SfM on-the-fly: Get better 3D from What You Capture

TL;DR

This paper introduces an improved real-time Structure from Motion method that enhances image matching, robustness, and collaborative reconstruction, resulting in more complete and accurate 3D models efficiently.

Contribution

The paper presents three key advancements: faster image matching with HNSW graphs, a self-adaptive weighting for robust local bundle adjustment, and multi-agent collaborative reconstruction.

Findings

More complete 3D reconstructions achieved

Faster processing times demonstrated

Enhanced robustness in diverse scenarios

Abstract

In the last twenty years, Structure from Motion (SfM) has been a constant research hotspot in the fields of photogrammetry, computer vision, robotics etc., whereas real-time performance is just a recent topic of growing interest. This work builds upon the original on-the-fly SfM (Zhan et al., 2024) and presents an updated version with three new advancements to get better 3D from what you capture: (i) real-time image matching is further boosted by employing the Hierarchical Navigable Small World (HNSW) graphs, thus more true positive overlapping image candidates are faster identified; (ii) a self-adaptive weighting strategy is proposed for robust hierarchical local bundle adjustment to improve the SfM results; (iii) multiple agents are included for supporting collaborative SfM and seamlessly merge multiple 3D reconstructions into a complete 3D scene when commonly registered images appear. Various comprehensive experiments demonstrate that the proposed SfM method (named on-the-fly SfMv2) can generate more complete and robust 3D reconstructions in a high time-efficient way. Code is available at http://yifeiyu225.github.io/on-the-flySfMv2.github.io/.