Rag Performance Prediction for Question Answering

TL;DR

This paper investigates methods to predict the performance gains of retrieval-augmented generation in question answering, introducing a novel supervised predictor that models semantic relationships for improved accuracy.

Contribution

It introduces a new supervised predictor that explicitly models semantic relationships, outperforming existing pre- and post-retrieval prediction methods.

Findings

The novel predictor achieves the best prediction quality among tested methods.

Post-generation predictors can effectively estimate RAG performance gains.

Semantic relationship modeling enhances prediction accuracy.

Abstract

We address the task of predicting the gain of using RAG (retrieval augmented generation) for question answering with respect to not using it. We study the performance of a few pre-retrieval and post-retrieval predictors originally devised for ad hoc retrieval. We also study a few post-generation predictors, one of which is novel to this study and posts the best prediction quality. Our results show that the most effective prediction approach is a novel supervised predictor that explicitly models the semantic relationships among the question, retrieved passages, and the generated answer.