Deploy DINO with Many-to-Many Association

TL;DR

This paper introduces Harmonic Consensus Maximization (HCM), a faster, more precise robust mechanism for many-to-many feature matching using DINO features, enhancing zero-shot generalization in image matching tasks.

Contribution

It proposes a novel likelihood-based approach and HCM mechanism to improve computational efficiency and accuracy in many-to-many feature association for zero-shot image matching.

Findings

HCM achieves faster and finer-grained robust matching.

DINO features with m-to-m association outperform specialized models on OOD datasets.

The approach enables zero-shot deployment without adaptation.

Abstract

Motivated by the limited generalization of supervised image matching models to unseen image domains, we explore the zero-shot deployment of DINO features for this task. The generalist visual representation extracted from DINO has inherent ambiguity when used to match feature points among semantically similar instances, prompting us to adopt a many-to-many (m-to-m) matching paradigm. However, the existing robust mechanism under m-to-m data association is computationally heavy, which requires finding a maximum-cardinality matching in the inlier association graph for each parameter evaluation. To address this inefficiency, we introduce a novel likelihood perspective, which interprets the existing method as a zeroth-order approximation of otherwise intractable likelihood calculation,and inspires us to propose a faster and finer-grained robust mechanism, termed as Harmonic Consensus Maximization (HCM). Take camera pose estimation as an exemplifying downstream task, we demonstrate that general-purpose visual features, used out of the box without any adaptation, can compete with specialized matching models on out-of-distribution datasets when mated with m-to-m association and the HCM mechanism.