Detector-Free Structure from Motion

TL;DR

This paper introduces a detector-free structure-from-motion framework that improves 3D reconstruction accuracy, especially in texture-poor scenes, by avoiding keypoint detection and employing iterative refinement with attention-based matching.

Contribution

The paper presents a novel detector-free SfM approach that combines coarse reconstruction with iterative refinement using attention mechanisms, outperforming traditional detector-based methods.

Findings

Outperforms existing detector-based SfM systems on benchmark datasets.

Successfully reconstructs texture-poor scenes.

Achieved first place in Image Matching Challenge 2023.

Abstract

We propose a new structure-from-motion framework to recover accurate camera poses and point clouds from unordered images. Traditional SfM systems typically rely on the successful detection of repeatable keypoints across multiple views as the first step, which is difficult for texture-poor scenes, and poor keypoint detection may break down the whole SfM system. We propose a new detector-free SfM framework to draw benefits from the recent success of detector-free matchers to avoid the early determination of keypoints, while solving the multi-view inconsistency issue of detector-free matchers. Specifically, our framework first reconstructs a coarse SfM model from quantized detector-free matches. Then, it refines the model by a novel iterative refinement pipeline, which iterates between an attention-based multi-view matching module to refine feature tracks and a geometry refinement module to improve the reconstruction accuracy. Experiments demonstrate that the proposed framework outperforms existing detector-based SfM systems on common benchmark datasets. We also collect a texture-poor SfM dataset to demonstrate the capability of our framework to reconstruct texture-poor scenes. Based on this framework, we take $\textit{first place}$ in Image Matching Challenge 2023.