Investigating the Robustness of Retrieval-Augmented Generation at the Query Level

TL;DR

This paper examines how small changes in input queries affect the performance of retrieval-augmented generation systems, highlighting their sensitivity and proposing a framework for robustness evaluation.

Contribution

It introduces a systematic evaluation framework for query-level robustness in RAG systems and provides extensive experimental analysis with practical recommendations.

Findings

Retrievers' performance degrades significantly with minor query perturbations

End-to-end RAG systems are highly sensitive to input variations

Proposed evaluation framework enables systematic robustness assessment

Abstract

Large language models (LLMs) are very costly and inefficient to update with new information. To address this limitation, retrieval-augmented generation (RAG) has been proposed as a solution that dynamically incorporates external knowledge during inference, improving factual consistency and reducing hallucinations. Despite its promise, RAG systems face practical challenges-most notably, a strong dependence on the quality of the input query for accurate retrieval. In this paper, we investigate the sensitivity of different components in the RAG pipeline to various types of query perturbations. Our analysis reveals that the performance of commonly used retrievers can degrade significantly even under minor query variations. We study each module in isolation as well as their combined effect in an end-to-end question answering setting, using both general-domain and domain-specific datasets. Additionally, we propose an evaluation framework to systematically assess the query-level robustness of RAG pipelines and offer actionable recommendations for practitioners based on the results of more than 1092 experiments we performed.