Attribute-Aware Implicit Modality Alignment for Text Attribute Person Search

TL;DR

This paper introduces a novel framework for text-based person search that effectively aligns local and global features between textual attributes and images, significantly improving retrieval accuracy by addressing modality gaps.

Contribution

The proposed Attribute-Aware Implicit Modality Alignment (AIMA) framework combines structured sentence prompts, a masked attribute prediction module, and an IoU-guided contrastive loss to enhance cross-modal alignment.

Findings

Outperforms state-of-the-art on Market-1501 Attribute, PETA, and PA100K datasets.

Effectively aligns local attribute features with image details.

Improves semantic consistency in attribute embedding space.

Abstract

Text attribute person search aims to find specific pedestrians through given textual attributes, which is very meaningful in the scene of searching for designated pedestrians through witness descriptions. The key challenge is the significant modality gap between textual attributes and images. Previous methods focused on achieving explicit representation and alignment through unimodal pre-trained models. Nevertheless, the absence of inter-modality correspondence in these models may lead to distortions in the local information of intra-modality. Moreover, these methods only considered the alignment of inter-modality and ignored the differences between different attribute categories. To mitigate the above problems, we propose an Attribute-Aware Implicit Modality Alignment (AIMA) framework to learn the correspondence of local representations between textual attributes and images and combine global representation matching to narrow the modality gap. Firstly, we introduce the CLIP model as the backbone and design prompt templates to transform attribute combinations into structured sentences. This facilitates the model's ability to better understand and match image details. Next, we design a Masked Attribute Prediction (MAP) module that predicts the masked attributes after the interaction of image and masked textual attribute features through multi-modal interaction, thereby achieving implicit local relationship alignment. Finally, we propose an Attribute-IoU Guided Intra-Modal Contrastive (A-IoU IMC) loss, aligning the distribution of different textual attributes in the embedding space with their IoU distribution, achieving better semantic arrangement. Extensive experiments on the Market-1501 Attribute, PETA, and PA100K datasets show that the performance of our proposed method significantly surpasses the current state-of-the-art methods.