Mind the Gap: Aligning Vision Foundation Models to Image Feature Matching

TL;DR

This paper introduces IMD, a framework that aligns vision foundation models with image feature matching by integrating generative diffusion models and a cross-image prompting mechanism, significantly improving multi-instance matching performance.

Contribution

The paper proposes a novel framework combining diffusion models and a cross-image prompting module to address misalignment in vision foundation models for feature matching.

Findings

IMD achieves state-of-the-art results on standard benchmarks.

12% improvement on the IMIM multi-instance benchmark.

Effectively mitigates the misalignment issue in feature matching.

Abstract

Leveraging the vision foundation models has emerged as a mainstream paradigm that improves the performance of image feature matching. However, previous works have ignored the misalignment when introducing the foundation models into feature matching. The misalignment arises from the discrepancy between the foundation models focusing on single-image understanding and the cross-image understanding requirement of feature matching. Specifically, 1) the embeddings derived from commonly used foundation models exhibit discrepancies with the optimal embeddings required for feature matching; 2) lacking an effective mechanism to leverage the single-image understanding ability into cross-image understanding. A significant consequence of the misalignment is they struggle when addressing multi-instance feature matching problems. To address this, we introduce a simple but effective framework, called IMD (Image feature Matching with a pre-trained Diffusion model) with two parts: 1) Unlike the dominant solutions employing contrastive-learning based foundation models that emphasize global semantics, we integrate the generative-based diffusion models to effectively capture instance-level details. 2) We leverage the prompt mechanism in generative model as a natural tunnel, propose a novel cross-image interaction prompting module to facilitate bidirectional information interaction between image pairs. To more accurately measure the misalignment, we propose a new benchmark called IMIM, which focuses on multi-instance scenarios. Our proposed IMD establishes a new state-of-the-art in commonly evaluated benchmarks, and the superior improvement 12% in IMIM indicates our method efficiently mitigates the misalignment.