Ambiguity-Aware and High-Order Relation Learning for Multi-Grained Image-Text Matching

TL;DR

This paper introduces AAHR, a novel framework that enhances multi-grained image-text matching by addressing semantic ambiguities and leveraging high-order relations through dynamic clustering, GNNs, and contrastive learning.

Contribution

It proposes a unified representation space and relation learning strategies that improve semantic understanding and discrimination in image-text matching tasks.

Findings

Outperforms state-of-the-art on Flickr30K, MSCOCO, ECCV Caption datasets.

Significantly improves matching accuracy and efficiency.

Effectively mitigates semantic ambiguities and leverages high-order relations.

Abstract

Image-text matching is crucial for bridging the semantic gap between computer vision and natural language processing. However, existing methods still face challenges in handling high-order associations and semantic ambiguities among similar instances. These ambiguities arise from subtle differences between soft positive samples (semantically similar but incorrectly labeled) and soft negative samples (locally matched but globally inconsistent), creating matching uncertainties. Furthermore, current methods fail to fully utilize the neighborhood relationships among semantically similar instances within training batches, limiting the model's ability to learn high-order shared knowledge. This paper proposes the Ambiguity-Aware and High-order Relation learning framework (AAHR) to address these issues. AAHR constructs a unified representation space through dynamic clustering prototype contrastive learning, effectively mitigating the soft positive sample problem. The framework introduces global and local feature extraction mechanisms and an adaptive aggregation network, significantly enhancing full-grained semantic understanding capabilities. Additionally, AAHR employs intra-modal and inter-modal correlation matrices to investigate neighborhood relationships among sample instances thoroughly. It incorporates GNN to enhance semantic interactions between instances. Furthermore, AAHR integrates momentum contrastive learning to expand the negative sample set. These combined strategies significantly improve the model's ability to discriminate between features. Experimental results demonstrate that AAHR outperforms existing state-of-the-art methods on Flickr30K, MSCOCO, and ECCV Caption datasets, considerably improving the accuracy and efficiency of image-text matching. The code and model checkpoints for this research are available at https://github.com/Image-Text-Matching/AAHR .