MIRACL-VISION: A Large, multilingual, visual document retrieval benchmark

TL;DR

MIRACL-VISION is a comprehensive multilingual benchmark designed to evaluate visual document retrieval models across 18 languages, addressing limitations of existing benchmarks and highlighting the performance gap between visual and text-based retrieval methods.

Contribution

It introduces MIRACL-VISION, a new multilingual visual document retrieval benchmark based on the MIRACL dataset, with a novel method to filter easy negatives for more challenging evaluation.

Findings

Visual models perform up to 59.7% worse than text models in multilingual retrieval.

Even in English, visual models lag behind text-based models by 12.1%.

MIRACL-VISION provides a challenging benchmark for developing robust visual retrieval models.

Abstract

Document retrieval is an important task for search and Retrieval-Augmented Generation (RAG) applications. Large Language Models (LLMs) have contributed to improving the accuracy of text-based document retrieval. However, documents with complex layout and visual elements like tables, charts and infographics are not perfectly represented in textual format. Recently, image-based document retrieval pipelines have become popular, which use visual large language models (VLMs) to retrieve relevant page images given a query. Current evaluation benchmarks on visual document retrieval are limited, as they primarily focus only English language, rely on synthetically generated questions and offer a small corpus size. Therefore, we introduce MIRACL-VISION, a multilingual visual document retrieval evaluation benchmark. MIRACL-VISION covers 18 languages, and is an extension of the MIRACL dataset, a popular benchmark to evaluate text-based multilingual retrieval pipelines. MIRACL was built using a human-intensive annotation process to generate high-quality questions. In order to reduce MIRACL-VISION corpus size to make evaluation more compute friendly while keeping the datasets challenging, we have designed a method for eliminating the "easy" negatives from the corpus. We conducted extensive experiments comparing MIRACL-VISION with other benchmarks, using popular public text and image models. We observe a gap in state-of-the-art VLM-based embedding models on multilingual capabilities, with up to 59.7% lower retrieval accuracy than a text-based retrieval models. Even for the English language, the visual models retrieval accuracy is 12.1% lower compared to text-based models. MIRACL-VISION is a challenging, representative, multilingual evaluation benchmark for visual retrieval pipelines and will help the community build robust models for document retrieval.