Retrieval or Representation? Reassessing Benchmark Gaps in Multilingual and Visually Rich RAG

TL;DR

This paper investigates whether improvements in multilingual and visually rich retrieval-augmented generation are due to better retrieval methods or improved document representations, highlighting the importance of evaluation decomposition.

Contribution

The study shows that document representation quality is the main factor behind benchmark improvements, emphasizing the need for separate evaluation of transcription and retrieval.

Findings

BM25 can close gaps in multilingual and visual benchmarks when combined with proper preprocessing.

Better document representation, rather than retrieval mechanism improvements, drives performance gains.

Decomposed benchmarks are necessary to accurately attribute progress in RAG systems.

Abstract

Retrieval-augmented generation (RAG) is a common way to ground language models in external documents and up-to-date information. Classical retrieval systems relied on lexical methods such as BM25, which rank documents by term overlap with corpus-level weighting. End-to-end multimodal retrievers trained on large query-document datasets claim substantial improvements over these approaches, especially for multilingual documents with complex visual layouts. We demonstrate that better document representation is the primary driver of benchmark improvements. By systematically varying transcription and preprocessing methods while holding the retrieval mechanism fixed, we demonstrate that BM25 can recover large gaps on multilingual and visual benchmarks. Our findings call for decomposed evaluation benchmarks that separately measure transcription and retrieval capabilities, enabling the field to correctly attribute progress and focus effort where it matters.