AMNS: Attention-Weighted Selective Mask and Noise Label Suppression for Text-to-Image Person Retrieval

TL;DR

This paper introduces a novel approach for text-to-image person retrieval that effectively mitigates noisy label issues and improves feature extraction by using attention-weighted masking and specialized loss functions, leading to better retrieval accuracy.

Contribution

It proposes a combined method with noise suppression, attention-weighted masking, and advanced loss functions to enhance robustness against noisy data in text-to-image retrieval.

Findings

Improved retrieval accuracy on benchmark datasets.

Effective mitigation of noisy correspondence issues.

Enhanced feature extraction through selective masking.

Abstract

Most existing text-to-image person retrieval methods usually assume that the training image-text pairs are perfectly aligned; however, the noisy correspondence(NC) issue (i.e., incorrect or unreliable alignment) exists due to poor image quality and labeling errors. Additionally, random masking augmentation may inadvertently discard critical semantic content, introducing noisy matches between images and text descriptions. To address the above two challenges, we propose a noise label suppression method to mitigate NC and an Attention-Weighted Selective Mask (AWM) strategy to resolve the issues caused by random masking. Specifically, the Bidirectional Similarity Distribution Matching (BSDM) loss enables the model to effectively learn from positive pairs while preventing it from over-relying on them, thereby mitigating the risk of overfitting to noisy labels. In conjunction with this, Weight Adjustment Focal (WAF) loss improves the model's ability to handle hard samples. Furthermore, AWM processes raw images through an EMA version of the image encoder, selectively retaining tokens with strong semantic connections to the text, enabling better feature extraction. Extensive experiments demonstrate the effectiveness of our approach in addressing noise-related issues and improving retrieval performance.