Relative Pose Estimation through Affine Corrections of Monocular Depth Priors

TL;DR

This paper introduces affine correction-based solvers for relative pose estimation using monocular depth priors, significantly improving accuracy over traditional keypoint methods across various datasets and conditions.

Contribution

It develops novel affine correction solvers for relative pose estimation that explicitly handle monocular depth ambiguities, integrating them into a hybrid pipeline with classic methods.

Findings

Large improvements over keypoint-based baselines.

Effective under both calibrated and uncalibrated conditions.

Consistent performance gains with different feature matchers and depth models.

Abstract

Monocular depth estimation (MDE) models have undergone significant advancements over recent years. Many MDE models aim to predict affine-invariant relative depth from monocular images, while recent developments in large-scale training and vision foundation models enable reasonable estimation of metric (absolute) depth. However, effectively leveraging these predictions for geometric vision tasks, in particular relative pose estimation, remains relatively under explored. While depths provide rich constraints for cross-view image alignment, the intrinsic noise and ambiguity from the monocular depth priors present practical challenges to improving upon classic keypoint-based solutions. In this paper, we develop three solvers for relative pose estimation that explicitly account for independent affine (scale and shift) ambiguities, covering both calibrated and uncalibrated conditions. We further propose a hybrid estimation pipeline that combines our proposed solvers with classic point-based solvers and epipolar constraints. We find that the affine correction modeling is beneficial to not only the relative depth priors but also, surprisingly, the "metric" ones. Results across multiple datasets demonstrate large improvements of our approach over classic keypoint-based baselines and PnP-based solutions, under both calibrated and uncalibrated setups. We also show that our method improves consistently with different feature matchers and MDE models, and can further benefit from very recent advances on both modules. Code is available at https://github.com/MarkYu98/madpose.