Invisible Relevance Bias: Text-Image Retrieval Models Prefer AI-Generated Images

TL;DR

This paper uncovers an invisible relevance bias in text-image retrieval models favoring AI-generated images over real ones, explores its causes, and proposes a debiasing method to mitigate this bias.

Contribution

The study constructs a benchmark to detect bias, demonstrates the bias's prevalence and exacerbation with AI-generated images, and introduces a training method to reduce the bias.

Findings

AI-generated images are ranked higher despite similar relevance.

Bias is consistent across different models and training data.

Debiasing reduces the ranking bias and reveals embedded information in generated images.

Abstract

With the advancement of generation models, AI-generated content (AIGC) is becoming more realistic, flooding the Internet. A recent study suggests that this phenomenon causes source bias in text retrieval for web search. Specifically, neural retrieval models tend to rank generated texts higher than human-written texts. In this paper, we extend the study of this bias to cross-modal retrieval. Firstly, we successfully construct a suitable benchmark to explore the existence of the bias. Subsequent extensive experiments on this benchmark reveal that AI-generated images introduce an invisible relevance bias to text-image retrieval models. Specifically, our experiments show that text-image retrieval models tend to rank the AI-generated images higher than the real images, even though the AI-generated images do not exhibit more visually relevant features to the query than real images. This invisible relevance bias is prevalent across retrieval models with varying training data and architectures. Furthermore, our subsequent exploration reveals that the inclusion of AI-generated images in the training data of the retrieval models exacerbates the invisible relevance bias. The above phenomenon triggers a vicious cycle, which makes the invisible relevance bias become more and more serious. To elucidate the potential causes of invisible relevance and address the aforementioned issues, we introduce an effective training method aimed at alleviating the invisible relevance bias. Subsequently, we apply our proposed debiasing method to retroactively identify the causes of invisible relevance, revealing that the AI-generated images induce the image encoder to embed additional information into their representation. This information exhibits a certain consistency across generated images with different semantics and can make the retriever estimate a higher relevance score.