Who Handles Orientation? Investigating Invariance in Feature Matching

TL;DR

This paper investigates where to incorporate rotation invariance in feature matching pipelines, finding that early descriptor-based invariance improves speed and robustness without sacrificing upright performance.

Contribution

It demonstrates that embedding rotation invariance in descriptors is as effective as in matchers and enables faster, more robust image matching across diverse datasets.

Findings

Incorporating rotation invariance in descriptors yields similar performance to matcher-based invariance.

Early descriptor invariance allows for faster rotation-invariant matchers.

Increasing training data size improves generalization to rotated images.

Abstract

Finding matching keypoints between images is a core problem in 3D computer vision. However, modern matchers struggle with large in-plane rotations. A straightforward mitigation is to learn rotation invariance via data augmentation. However, it remains unclear at which stage rotation invariance should be incorporated. In this paper, we study this in the context of a modern sparse matching pipeline. We perform extensive experiments by training on a large collection of 3D vision datasets and evaluating on popular image matching benchmarks. Surprisingly, we find that incorporating rotation invariance already in the descriptor yields similar performance to handling it in the matcher. However, rotation invariance is achieved earlier in the matcher when it is learned in the descriptor, allowing for a faster rotation-invariant matcher. Further, we find that enforcing rotation invariance does not hurt upright performance when trained at scale. Finally, we study the emergence of rotation invariance through scale and find that increasing the training data size substantially improves generalization to rotated images. We release two matchers robust to in-plane rotations that achieve state-of-the-art performance on e.g. multi-modal (WxBS), extreme (HardMatch), and satellite image matching (SatAst). Code is available at https://github.com/davnords/loma.