PRGB Benchmark: A Robust Placeholder-Assisted Algorithm for Benchmarking Retrieval-Augmented Generation

TL;DR

The PRGB Benchmark introduces a detailed, multi-level evaluation framework for RAG systems, focusing on LLM-specific capabilities and the role of external knowledge, to improve reliability and efficiency.

Contribution

It presents a novel placeholder-based evaluation approach and a comprehensive benchmark for assessing LLMs in RAG systems at multiple granular levels.

Findings

Current LLMs show limitations in error resilience and context faithfulness in RAG.

The benchmark reveals specific weaknesses in representative LLMs' generation capabilities.

The framework enables systematic, reproducible evaluation of RAG system components.

Abstract

Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by integrating external knowledge, where the LLM's ability to generate responses based on the combination of a given query and retrieved documents is crucial. However, most benchmarks focus on overall RAG system performance, rarely assessing LLM-specific capabilities. Current benchmarks emphasize broad aspects such as noise robustness, but lack a systematic and granular evaluation framework on document utilization. To this end, we introduce \textit{Placeholder-RAG-Benchmark}, a multi-level fine-grained benchmark, emphasizing the following progressive dimensions: (1) multi-level filtering abilities, (2) combination abilities, and (3) reference reasoning. To provide a more nuanced understanding of LLMs' roles in RAG systems, we formulate an innovative placeholder-based approach to decouple the contributions of the LLM's parametric knowledge and the external knowledge. Experiments demonstrate the limitations of representative LLMs in the RAG system's generation capabilities, particularly in error resilience and context faithfulness. Our benchmark provides a reproducible framework for developing more reliable and efficient RAG systems. Our code is available in https://github.com/Alipay-Med/PRGB.