ITSELF: Attention Guided Fine-Grained Alignment for Vision-Language Retrieval

TL;DR

ITSELF introduces an attention-guided framework that leverages model attention to improve fine-grained alignment in vision-language retrieval, achieving state-of-the-art results without extra supervision.

Contribution

The paper proposes a novel attention-guided implicit local alignment method using GRAB, MARS, and ATS to enhance TBPS performance and robustness.

Findings

Achieves state-of-the-art results on three TBPS benchmarks.

Demonstrates strong cross-dataset generalization.

Effective without additional prior supervision.

Abstract

Vision Language Models (VLMs) have rapidly advanced and show strong promise for text-based person search (TBPS), a task that requires capturing fine-grained relationships between images and text to distinguish individuals. Previous methods address these challenges through local alignment, yet they are often prone to shortcut learning and spurious correlations, yielding misalignment. Moreover, injecting prior knowledge can distort intra-modality structure. Motivated by our finding that encoder attention surfaces spatially precise evidence from the earliest training epochs, and to alleviate these issues, we introduceITSELF, an attention-guided framework for implicit local alignment. At its core, Guided Representation with Attentive Bank (GRAB) converts the model's own attention into an Attentive Bank of high-saliency tokens and applies local objectives on this bank, learning fine-grained correspondences without extra supervision. To make the selection reliable and non-redundant, we introduce Multi-Layer Attention for Robust Selection (MARS), which aggregates attention across layers and performs diversity-aware top-k selection; and Adaptive Token Scheduler (ATS), which schedules the retention budget from coarse to fine over training, preserving context early while progressively focusing on discriminative details. Extensive experiments on three widely used TBPS benchmarks showstate-of-the-art performance and strong cross-dataset generalization, confirming the effectiveness and robustness of our approach without additional prior supervision. Our project is publicly available at https://trhuuloc.github.io/itself