Joint Graph Learning and Matching for Semantic Feature Correspondence

TL;DR

This paper introduces GLAM, a novel attention-based joint graph learning and matching network that automatically discovers reliable graph structures to significantly improve semantic feature matching performance.

Contribution

GLAM employs a pure attention framework with self- and cross-attention mechanisms for joint graph learning and matching, eliminating reliance on heuristically generated graph patterns.

Findings

Outperforms state-of-the-art methods on Pascal VOC, Willow Object, and SPair-71k benchmarks.

Learnt graph patterns enhance existing deep graph matching methods.

Achieves significant margin improvements across all evaluated benchmarks.

Abstract

In recent years, powered by the learned discriminative representation via graph neural network (GNN) models, deep graph matching methods have made great progresses in the task of matching semantic features. However, these methods usually rely on heuristically generated graph patterns, which may introduce unreliable relationships to hurt the matching performance. In this paper, we propose a joint \emph{graph learning and matching} network, named GLAM, to explore reliable graph structures for boosting graph matching. GLAM adopts a pure attention-based framework for both graph learning and graph matching. Specifically, it employs two types of attention mechanisms, self-attention and cross-attention for the task. The self-attention discovers the relationships between features and to further update feature representations over the learnt structures; and the cross-attention computes cross-graph correlations between the two feature sets to be matched for feature reconstruction. Moreover, the final matching solution is directly derived from the output of the cross-attention layer, without employing a specific matching decision module. The proposed method is evaluated on three popular visual matching benchmarks (Pascal VOC, Willow Object and SPair-71k), and it outperforms previous state-of-the-art graph matching methods by significant margins on all benchmarks. Furthermore, the graph patterns learnt by our model are validated to be able to remarkably enhance previous deep graph matching methods by replacing their handcrafted graph structures with the learnt ones.