Predicting Retrieval Utility and Answer Quality in Retrieval-Augmented Generation

TL;DR

This paper introduces methods to predict the usefulness of retrieved documents and the quality of answers in retrieval-augmented generation, improving the ability to estimate RAG performance using various features.

Contribution

It proposes new prediction tasks (RPP and GPP) and combines multiple feature categories to enhance prediction accuracy in RAG systems.

Findings

Combining multiple feature categories improves prediction accuracy.

Reader-centric features like perplexity enhance predictions.

Topical relevance correlates with retrieval utility.

Abstract

The quality of answers generated by large language models (LLMs) in retrieval-augmented generation (RAG) is largely influenced by the contextual information contained in the retrieved documents. A key challenge for improving RAG is to predict both the utility of retrieved documents -- quantified as the performance gain from using context over generation without context -- and the quality of the final answers in terms of correctness and relevance. In this paper, we define two prediction tasks within RAG. The first is retrieval performance prediction (RPP), which estimates the utility of retrieved documents. The second is generation performance prediction (GPP), which estimates the final answer quality. We hypothesise that in RAG, the topical relevance of retrieved documents correlates with their utility, suggesting that query performance prediction (QPP) approaches can be adapted for RPP and GPP. Beyond these retriever-centric signals, we argue that reader-centric features, such as the LLM's perplexity of the retrieved context conditioned on the input query, can further enhance prediction accuracy for both RPP and GPP. Finally, we propose that features reflecting query-agnostic document quality and readability can also provide useful signals to the predictions. We train linear regression models with the above categories of predictors for both RPP and GPP. Experiments on the Natural Questions (NQ) dataset show that combining predictors from multiple feature categories yields the most accurate estimates of RAG performance.