Unanswerability Evaluation for Retrieval Augmented Generation

TL;DR

This paper introduces UAEval4RAG, a framework for evaluating how well retrieval-augmented generation systems can reject unanswerable queries, addressing a gap in existing evaluation methods.

Contribution

We propose a new evaluation framework with a taxonomy of unanswerable categories and automatic query synthesis, enabling comprehensive assessment of RAG system robustness.

Findings

Component choice significantly impacts performance trade-offs.

Prompt design influences the balance between answer accuracy and rejection rate.

Our framework reveals hidden performance trade-offs in RAG systems.

Abstract

Existing evaluation frameworks for retrieval-augmented generation (RAG) systems focus on answerable queries, but they overlook the importance of appropriately rejecting unanswerable requests. In this paper, we introduce UAEval4RAG, a framework designed to evaluate whether RAG systems can handle unanswerable queries effectively. We define a taxonomy with six unanswerable categories, and UAEval4RAG automatically synthesizes diverse and challenging queries for any given knowledge base with unanswered ratio and acceptable ratio metrics. We conduct experiments with various RAG components, including retrieval models, rewriting methods, rerankers, language models, and prompting strategies, and reveal hidden trade-offs in performance of RAG systems. Our findings highlight the critical role of component selection and prompt design in optimizing RAG systems to balance the accuracy of answerable queries with high rejection rates of unanswerable ones. UAEval4RAG provides valuable insights and tools for developing more robust and reliable RAG systems.