RegionRAG: Region-level Retrieval-Augmented Generation for Visual Document Understanding

TL;DR

RegionRAG introduces a region-level retrieval approach for visual document understanding, significantly enhancing retrieval precision and answer accuracy by focusing on relevant visual regions rather than entire documents.

Contribution

It proposes a novel region-level retrieval framework with hybrid supervision and dynamic grouping, improving efficiency and accuracy over document-level methods.

Findings

Achieves 10.02% higher R@1 retrieval accuracy on average.

Boosts question answering accuracy by 3.56%.

Uses only 71.42% of visual tokens compared to prior methods.

Abstract

Multi-modal Retrieval-Augmented Generation (RAG) has become a critical method for empowering LLMs by leveraging candidate visual documents. However, current methods consider the entire document as the basic retrieval unit, introducing substantial irrelevant visual content in two ways: 1) Relevant documents often contain large regions unrelated to the query, diluting the focus on salient information; 2) Retrieving multiple documents to increase recall further introduces redundant and irrelevant documents. These redundant contexts distract the model's attention and further degrade the performance. To address this challenge, we propose RegionRAG, a novel framework that shifts the retrieval paradigm from the document level to the region level. During training, we design a hybrid supervision strategy from both labeled data and unlabeled data to pinpoint relevant patches. During inference, we propose a dynamic pipeline that intelligently groups salient patches into complete semantic regions. By delegating the task of identifying relevant regions to the retriever, RegionRAG enables the generator to focus solely on concise, query-relevant visual content, improving both efficiency and accuracy. Experiments on six benchmarks demonstrate that RegionRAG achieves state-of-the-art performance. It improves retrieval accuracy by 10.02% in R@1 on average, and boosts question answering accuracy by 3.56% while using only 71.42% visual tokens compared with prior methods.