Benchmark Granularity and Model Robustness for Image-Text Retrieval

TL;DR

This paper investigates how dataset granularity and query perturbations influence the performance and robustness of Vision-Language Models in image-text retrieval, revealing that richer captions improve accuracy and that word order critically affects model behavior.

Contribution

It introduces a detailed analysis of dataset granularity and perturbations, highlighting their impact on model robustness and performance in image-text retrieval tasks.

Findings

Richer captions significantly improve retrieval accuracy, especially in text-to-image tasks.

Perturbations generally degrade performance but can sometimes unexpectedly enhance retrieval.

Word order is a critical factor affecting model robustness, contrary to prior assumptions.

Abstract

Image-Text Retrieval (ITR) systems are central to multimodal information access, with Vision-Language Models (VLMs) showing strong performance on standard benchmarks. However, these benchmarks predominantly rely on coarse-grained annotations, limiting their ability to reveal how models perform under real-world conditions, where query granularity varies. Motivated by this gap, we examine how dataset granularity and query perturbations affect retrieval performance and robustness across four architecturally diverse VLMs (ALIGN, AltCLIP, CLIP, and GroupViT). Using both standard benchmarks (MS-COCO, Flickr30k) and their fine-grained variants, we show that richer captions consistently enhance retrieval, especially in text-to-image tasks, where we observe an average improvement of 16.23%, compared to 6.44% in image-to-text. To assess robustness, we introduce a taxonomy of perturbations and conduct extensive experiments, revealing that while perturbations typically degrade performance, they can also unexpectedly improve retrieval, exposing nuanced model behaviors. Notably, word order emerges as a critical factor -- contradicting prior assumptions of model insensitivity to it. Our results highlight variation in model robustness and a dataset-dependent relationship between caption granularity and perturbation sensitivity and emphasize the necessity of evaluating models on datasets of varying granularity.