Dual-Granularity Cross-Modal Identity Association for Weakly-Supervised Text-to-Person Image Matching

TL;DR

This paper introduces a dual-granularity identity association mechanism for weakly-supervised text-to-person image matching, improving the model's ability to handle complex identity relationships with enhanced accuracy and robustness.

Contribution

It proposes a novel local-and-global dual-granularity association mechanism and a confidence-based dynamic adjustment network for better cross-modal identity matching.

Findings

Significant boost in matching accuracy

Effective handling of weakly associated samples

Enhanced robustness through novel sample construction

Abstract

Weakly supervised text-to-person image matching, as a crucial approach to reducing models' reliance on large-scale manually labeled samples, holds significant research value. However, existing methods struggle to predict complex one-to-many identity relationships, severely limiting performance improvements. To address this challenge, we propose a local-and-global dual-granularity identity association mechanism. Specifically, at the local level, we explicitly establish cross-modal identity relationships within a batch, reinforcing identity constraints across different modalities and enabling the model to better capture subtle differences and correlations. At the global level, we construct a dynamic cross-modal identity association network with the visual modality as the anchor and introduce a confidence-based dynamic adjustment mechanism, effectively enhancing the model's ability to identify weakly associated samples while improving overall sensitivity. Additionally, we propose an information-asymmetric sample pair construction method combined with consistency learning to tackle hard sample mining and enhance model robustness. Experimental results demonstrate that the proposed method substantially boosts cross-modal matching accuracy, providing an efficient and practical solution for text-to-person image matching.