MVAM: Multi-View Attention Method for Fine-grained Image-Text Matching

TL;DR

This paper introduces MVAM, a multi-view attention approach that improves image-text matching by capturing diverse, fine-grained details through multiple attention heads with a diversity objective, leading to better retrieval accuracy.

Contribution

The paper proposes a novel multi-view attention method with diversity constraints to enhance two-stream models for more comprehensive image-text representations.

Findings

Improved performance on MSCOCO and Flickr30K datasets.

Attention heads focus on distinct content aspects.

Enhanced fine-grained matching accuracy.

Abstract

Existing two-stream models, such as CLIP, encode images and text through independent representations, showing good performance while ensuring retrieval speed, have attracted attention from industry and academia. However, the single representation often struggles to capture complex content fully. Such models may ignore fine-grained information during matching, resulting in suboptimal retrieval results. To overcome this limitation and enhance the performance of two-stream models, we propose a Multi-view Attention Method (MVAM) for image-text matching. This approach leverages diverse attention heads with unique view codes to learn multiple representations for images and text, which are then concatenated for matching. We also incorporate a diversity objective to explicitly encourage attention heads to focus on distinct aspects of the input data, capturing complementary fine-grained details. This diversity enables the model to represent image-text pairs from multiple perspectives, ensuring a more comprehensive understanding and alignment of critical content. Our method allows models to encode images and text from different perspectives and focus on more critical details, leading to better matching performance. Our experiments on MSCOCO and Flickr30K demonstrate enhancements over existing models, and further case studies reveal that different attention heads can focus on distinct content, achieving more comprehensive representations.