Marginalized Bundle Adjustment: Multi-View Camera Pose from Monocular Depth Estimates

TL;DR

This paper introduces Marginalized Bundle Adjustment (MBA), a novel method that effectively integrates monocular depth estimates into Structure-from-Motion, achieving state-of-the-art results in multi-view 3D reconstruction and camera relocalization.

Contribution

The paper proposes MBA, a new approach that reduces MDE error variance in SfM, enabling the use of dense monocular depth maps for accurate multi-view 3D reconstruction.

Findings

MBA achieves state-of-the-art results in SfM and relocalization.

The method demonstrates robust performance across various scales.

MDE can be effectively integrated into SfM with MBA.

Abstract

Structure-from-Motion (SfM) is a fundamental 3D vision task for recovering camera parameters and scene geometry from multi-view images. While recent deep learning advances enable accurate Monocular Depth Estimation (MDE) from single images without depending on camera motion, integrating MDE into SfM remains a challenge. Unlike conventional triangulated sparse point clouds, MDE produces dense depth maps with significantly higher error variance. Inspired by modern RANSAC estimators, we propose Marginalized Bundle Adjustment (MBA) to mitigate MDE error variance leveraging its density. With MBA, we show that MDE depth maps are sufficiently accurate to yield SoTA or competitive results in SfM and camera relocalization tasks. Through extensive evaluations, we demonstrate consistently robust performance across varying scales, ranging from few-frame setups to large multi-view systems with thousands of images. Our method highlights the significant potential of MDE in multi-view 3D vision.