Evaluating Quality of Answers for Retrieval-Augmented Generation: A Strong LLM Is All You Need

TL;DR

This paper introduces vRAG-Eval, a new grading system for assessing answer quality in Retrieval-Augmented Generation, demonstrating that GPT-4 can reliably evaluate answers with high agreement to human judgments, especially in factual, closed-domain contexts.

Contribution

The paper presents vRAG-Eval, a novel evaluation framework for RAG answer quality, and shows GPT-4's evaluations align closely with human experts, highlighting LLMs as effective evaluators.

Findings

vRAG-Eval effectively assesses correctness, completeness, and honesty.

GPT-4 achieves 83% agreement with human judgments.

LLMs can serve as reliable evaluators in resource-intensive settings.

Abstract

We present a comprehensive study of answer quality evaluation in Retrieval-Augmented Generation (RAG) applications using vRAG-Eval, a novel grading system that is designed to assess correctness, completeness, and honesty. We further map the grading of quality aspects aforementioned into a binary score, indicating an accept or reject decision, mirroring the intuitive "thumbs-up" or "thumbs-down" gesture commonly used in chat applications. This approach suits factual business contexts where a clear decision opinion is essential. Our assessment applies vRAG-Eval to two Large Language Models (LLMs), evaluating the quality of answers generated by a vanilla RAG application. We compare these evaluations with human expert judgments and find a substantial alignment between GPT-4's assessments and those of human experts, reaching 83% agreement on accept or reject decisions. This study highlights the potential of LLMs as reliable evaluators in closed-domain, closed-ended settings, particularly when human evaluations require significant resources.