A Model for Multi-View Residual Covariances based on Perspective Deformation

TL;DR

This paper introduces a novel covariance model for multi-view residuals in SfM, odometry, and SLAM that accounts for perspective deformation effects, enhancing accuracy in visual state estimation methods.

Contribution

The main contribution is the derivation of a perspective deformation term for local 2D patches, improving residual covariance modeling in multi-view geometry.

Findings

Enhanced accuracy in feature-based and direct methods.

Validated model with synthetic and real data.

Improved state entropy estimation and visibility thresholds.

Abstract

In this work, we derive a model for the covariance of the visual residuals in multi-view SfM, odometry and SLAM setups. The core of our approach is the formulation of the residual covariances as a combination of geometric and photometric noise sources. And our key novel contribution is the derivation of a term modelling how local 2D patches suffer from perspective deformation when imaging 3D surfaces around a point. Together, these add up to an efficient and general formulation which not only improves the accuracy of both feature-based and direct methods, but can also be used to estimate more accurate measures of the state entropy and hence better founded point visibility thresholds. We validate our model with synthetic and real data and integrate it into photometric and feature-based Bundle Adjustment, improving their accuracy with a negligible overhead.