Who is in the Spotlight: The Hidden Bias Undermining Multimodal Retrieval-Augmented Generation

TL;DR

This paper investigates how the position of retrieved evidence affects the performance of multimodal Retrieval-Augmented Generation systems, revealing a consistent bias that impacts stability and fairness, and proposing methods to analyze and mitigate it.

Contribution

It provides the first comprehensive analysis of position bias in multimodal RAG systems, introducing the Position Sensitivity Index and visualization tools for bias detection.

Findings

Position bias follows a U-shaped accuracy curve.

Multimodal interactions amplify position bias.

Bias increases logarithmically with retrieval range.

Abstract

Multimodal Retrieval-Augmented Generation (RAG) systems have become essential in knowledge-intensive and open-domain tasks. As retrieval complexity increases, ensuring the robustness of these systems is critical. However, current RAG models are highly sensitive to the order in which evidence is presented, often resulting in unstable performance and biased reasoning, particularly as the number of retrieved items or modality diversity grows. This raises a central question: How does the position of retrieved evidence affect multimodal RAG performance? To answer this, we present the first comprehensive study of position bias in multimodal RAG systems. Through controlled experiments across text-only, image-only, and mixed-modality tasks, we observe a consistent U-shaped accuracy curve with respect to evidence position. To quantify this bias, we introduce the Position Sensitivity Index ($PSI_p$) and develop a visualization framework to trace attention allocation patterns across decoder layers. Our results reveal that multimodal interactions intensify position bias compared to unimodal settings, and that this bias increases logarithmically with retrieval range. These findings offer both theoretical and empirical foundations for position-aware analysis in RAG, highlighting the need for evidence reordering or debiasing strategies to build more reliable and equitable generation systems.