Visual-RAG: Benchmarking Text-to-Image Retrieval Augmented Generation for Visual Knowledge Intensive Queries

TL;DR

Visual-RAG introduces a benchmark for evaluating how effectively multimodal language models utilize retrieved images for visually grounded, knowledge-intensive question answering, revealing current limitations and areas for improvement.

Contribution

We present Visual-RAG, a novel benchmark that isolates and measures the contribution of retrieved images in multimodal RAG systems for visual knowledge questions.

Findings

Images significantly aid answer generation in Visual-RAG.

State-of-the-art models struggle to effectively utilize visual evidence.

Current models need better visual retrieval and grounding mechanisms.

Abstract

Retrieval-augmented generation (RAG) is a paradigm that augments large language models (LLMs) with external knowledge to tackle knowledge-intensive question answering. While several benchmarks evaluate Multimodal LLMs (MLLMs) under Multimodal RAG settings, they predominantly retrieve from textual corpora and do not explicitly assess how models exploit visual evidence during generation. Consequently, there still lacks benchmark that isolates and measures the contribution of retrieved images in RAG. We introduce Visual-RAG, a question-answering benchmark that targets visually grounded, knowledge-intensive questions. Unlike prior work, Visual-RAG requires text-to-image retrieval and the integration of retrieved clue images to extract visual evidence for answer generation. With Visual-RAG, we evaluate 5 open-source and 3 proprietary MLLMs, showcasing that images provide strong evidence in augmented generation. However, even state-of-the-art models struggle to efficiently extract and utilize visual knowledge. Our results highlight the need for improved visual retrieval, grounding, and attribution in multimodal RAG systems.