On-the-Fly SfM: What you capture is What you get

TL;DR

This paper introduces an online Structure from Motion (SfM) system that estimates camera poses and sparse point clouds in real-time during image capture, enabling immediate 3D reconstruction.

Contribution

It presents a novel on-the-fly SfM approach combining unsupervised global feature-based image retrieval, robust feature matching, and hierarchical local bundle adjustment for real-time performance.

Findings

Achieves robust online image registration during capture

Uses unsupervised global features for fast image retrieval

Employs hierarchical local bundle adjustment for efficiency

Abstract

Over the last decades, ample achievements have been made on Structure from motion (SfM). However, the vast majority of them basically work in an offline manner, i.e., images are firstly captured and then fed together into a SfM pipeline for obtaining poses and sparse point cloud. In this work, on the contrary, we present an on-the-fly SfM: running online SfM while image capturing, the newly taken On-the-Fly image is online estimated with the corresponding pose and points, i.e., what you capture is what you get. Specifically, our approach firstly employs a vocabulary tree that is unsupervised trained using learning-based global features for fast image retrieval of newly fly-in image. Then, a robust feature matching mechanism with least squares (LSM) is presented to improve image registration performance. Finally, via investigating the influence of newly fly-in image's connected neighboring images, an efficient hierarchical weighted local bundle adjustment (BA) is used for optimization. Extensive experimental results demonstrate that on-the-fly SfM can meet the goal of robustly registering the images while capturing in an online way.