ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios

TL;DR

ViDoRe v3 introduces a comprehensive multimodal benchmark for retrieval-augmented generation, emphasizing complex real-world scenarios involving visual elements, multi-document synthesis, and multilingual queries, to evaluate and improve state-of-the-art models.

Contribution

The paper presents ViDoRe v3, a new benchmark with diverse datasets, extensive annotations, and evaluation protocols for multimodal RAG in complex, real-world contexts.

Findings

Visual retrievers outperform textual ones.

Late-interaction models and reranking improve performance.

Hybrid and visual-only contexts enhance answer quality.

Abstract

Retrieval-Augmented Generation (RAG) pipelines must address challenges beyond simple single-document retrieval, such as interpreting visual elements (tables, charts, images), synthesizing information across documents, and providing accurate source grounding. Existing benchmarks fail to capture this complexity, often focusing on textual data, single-document comprehension, or evaluating retrieval and generation in isolation. We introduce ViDoRe v3, a comprehensive multimodal RAG benchmark featuring multi-type queries over visually rich document corpora. It covers 10 datasets across diverse professional domains, comprising ~26,000 document pages paired with 3,099 human-verified queries, each available in 6 languages. Through 12,000 hours of human annotation effort, we provide high-quality annotations for retrieval relevance, bounding box localization, and verified reference answers. Our evaluation of state-of-the-art RAG pipelines reveals that visual retrievers outperform textual ones, late-interaction models and textual reranking substantially improve performance, and hybrid or purely visual contexts enhance answer generation quality. However, current models still struggle with non-textual elements, open-ended queries, and fine-grained visual grounding. To encourage progress in addressing these challenges, the benchmark is released under a commercially permissive license at https://hf.co/vidore.