Building Rome with Convex Optimization

TL;DR

This paper introduces a scalable convex optimization approach for global bundle adjustment in structure from motion, leveraging learned depth and SDP relaxations to achieve fast, accurate, and initialization-free 3D reconstruction.

Contribution

It proposes a novel scaled bundle adjustment formulation with a tight convex SDP relaxation and an efficient solver, enabling scalable and certifiably optimal SfM pipelines.

Findings

XM-SfM outperforms existing methods in reconstruction quality.

The approach is significantly faster and more scalable.

The method is initialization-free and certifiably optimal.

Abstract

Global bundle adjustment is made easy by depth prediction and convex optimization. We (i) propose a scaled bundle adjustment (SBA) formulation that lifts 2D keypoint measurements to 3D with learned depth, (ii) design an empirically tight convex semidfinite program (SDP) relaxation that solves SBA to certfiable global optimality, (iii) solve the SDP relaxations at extreme scale with Burer-Monteiro factorization and a CUDA-based trust-region Riemannian optimizer (dubbed XM), (iv) build a structure from motion (SfM) pipeline with XM as the optimization engine and show that XM-SfM compares favorably with existing pipelines in terms of reconstruction quality while being significantly faster, more scalable, and initialization-free.