Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?

TL;DR

This paper introduces Double-Bench, a comprehensive, multilingual, multimodal evaluation system for document RAG systems, addressing current evaluation limitations and revealing key insights into model performance and over-confidence issues.

Contribution

The paper presents Double-Bench, a large-scale, fine-grained evaluation framework for document RAG systems, incorporating diverse data and human-verified queries to improve assessment accuracy.

Findings

Embedding models are converging in performance.

Current RAG frameworks often overestimate evidence support.

Stronger retrieval models are needed for better document understanding.

Abstract

Retrieval-Augmented Generation (RAG) systems using Multimodal Large Language Models (MLLMs) show great promise for complex document understanding, yet their development is critically hampered by inadequate evaluation. Current benchmarks often focus on specific part of document RAG system and use synthetic data with incomplete ground truth and evidence labels, therefore failing to reflect real-world bottlenecks and challenges. To overcome these limitations, we introduce Double-Bench: a new large-scale, multilingual, and multimodal evaluation system that is able to produce fine-grained assessment to each component within document RAG systems. It comprises 3,276 documents (72,880 pages) and 5,168 single- and multi-hop queries across 6 languages and 4 document types with streamlined dynamic update support for potential data contamination issues. Queries are grounded in exhaustively scanned evidence pages and verified by human experts to ensure maximum quality and completeness. Our comprehensive experiments across 9 state-of-the-art embedding models, 4 MLLMs and 4 end-to-end document RAG frameworks demonstrate the gap between text and visual embedding models is narrowing, highlighting the need in building stronger document retrieval models. Our findings also reveal the over-confidence dilemma within current document RAG frameworks that tend to provide answer even without evidence support. We hope our fully open-source Double-Bench provide a rigorous foundation for future research in advanced document RAG systems. We plan to retrieve timely corpus and release new benchmarks on an annual basis.