Knowing Where to Focus: Attention-Guided Alignment for Text-based Person Search

TL;DR

This paper introduces an Attention-Guided Alignment framework for Text-Based Person Search that dynamically masks meaningful words and enriches text descriptions to improve cross-modal representation alignment, achieving state-of-the-art results.

Contribution

The paper proposes a novel Attention-Guided Mask and Text Enrichment Module to enhance cross-modal alignment and handle noisy descriptions in TBPS.

Findings

Achieved new state-of-the-art Rank-1 accuracy on three benchmarks.

Effectively masks semantically meaningful words to improve alignment.

Enriches text descriptions to mitigate noise and improve robustness.

Abstract

In the realm of Text-Based Person Search (TBPS), mainstream methods aim to explore more efficient interaction frameworks between text descriptions and visual data. However, recent approaches encounter two principal challenges. Firstly, the widely used random-based Masked Language Modeling (MLM) considers all the words in the text equally during training. However, massive semantically vacuous words ('with', 'the', etc.) be masked fail to contribute efficient interaction in the cross-modal MLM and hampers the representation alignment. Secondly, manual descriptions in TBPS datasets are tedious and inevitably contain several inaccuracies. To address these issues, we introduce an Attention-Guided Alignment (AGA) framework featuring two innovative components: Attention-Guided Mask (AGM) Modeling and Text Enrichment Module (TEM). AGM dynamically masks semantically meaningful words by aggregating the attention weight derived from the text encoding process, thereby cross-modal MLM can capture information related to the masked word from text context and images and align their representations. Meanwhile, TEM alleviates low-quality representations caused by repetitive and erroneous text descriptions by replacing those semantically meaningful words with MLM's prediction. It not only enriches text descriptions but also prevents overfitting. Extensive experiments across three challenging benchmarks demonstrate the effectiveness of our AGA, achieving new state-of-the-art results with Rank-1 accuracy reaching 78.36%, 67.31%, and 67.4% on CUHK-PEDES, ICFG-PEDES, and RSTPReid, respectively.