Co-Attention for Conditioned Image Matching

TL;DR

This paper introduces a co-attention mechanism for conditioned image matching that improves correspondence accuracy under challenging conditions like large viewpoint and illumination changes, achieving state-of-the-art results.

Contribution

The paper presents a novel co-attention module (CoAM) that conditions features on both images, enhancing matching performance in difficult scenarios.

Findings

Significant performance gains on local matching tasks

Improved camera localization accuracy

Enhanced 3D reconstruction results

Abstract

We propose a new approach to determine correspondences between image pairs in the wild under large changes in illumination, viewpoint, context, and material. While other approaches find correspondences between pairs of images by treating the images independently, we instead condition on both images to implicitly take account of the differences between them. To achieve this, we introduce (i) a spatial attention mechanism (a co-attention module, CoAM) for conditioning the learned features on both images, and (ii) a distinctiveness score used to choose the best matches at test time. CoAM can be added to standard architectures and trained using self-supervision or supervised data, and achieves a significant performance improvement under hard conditions, e.g. large viewpoint changes. We demonstrate that models using CoAM achieve state of the art or competitive results on a wide range of tasks: local matching, camera localization, 3D reconstruction, and image stylization.