Evaluating Retrieval Quality in Retrieval-Augmented Generation

TL;DR

This paper introduces eRAG, a novel evaluation method for retrieval-augmented generation that correlates better with downstream performance and is more computationally efficient than traditional methods.

Contribution

eRAG provides a new evaluation framework that assesses retrieval quality based on downstream task performance, reducing computational costs and improving correlation with actual RAG system effectiveness.

Findings

eRAG achieves higher correlation with downstream performance (Kendall's τ up to 0.494).

eRAG significantly reduces GPU memory usage and runtime.

The method is effective across diverse datasets.

Abstract

Evaluating retrieval-augmented generation (RAG) presents challenges, particularly for retrieval models within these systems. Traditional end-to-end evaluation methods are computationally expensive. Furthermore, evaluation of the retrieval model's performance based on query-document relevance labels shows a small correlation with the RAG system's downstream performance. We propose a novel evaluation approach, eRAG, where each document in the retrieval list is individually utilized by the large language model within the RAG system. The output generated for each document is then evaluated based on the downstream task ground truth labels. In this manner, the downstream performance for each document serves as its relevance label. We employ various downstream task metrics to obtain document-level annotations and aggregate them using set-based or ranking metrics. Extensive experiments on a wide range of datasets demonstrate that eRAG achieves a higher correlation with downstream RAG performance compared to baseline methods, with improvements in Kendall's $\tau$ correlation ranging from 0.168 to 0.494. Additionally, eRAG offers significant computational advantages, improving runtime and consuming up to 50 times less GPU memory than end-to-end evaluation.