MLDocRAG: Multimodal Long-Context Document Retrieval Augmented Generation

TL;DR

MLDocRAG introduces a novel framework for multimodal long-context document retrieval and question answering, using a graph-based query-centric approach to improve evidence aggregation and answer accuracy across diverse modalities and pages.

Contribution

It proposes a Multimodal Chunk-Query Graph (MCQG) for organizing and retrieving multimodal content, advancing long-context understanding in multimodal QA tasks.

Findings

Improves retrieval quality and answer accuracy on benchmark datasets

Effectively aggregates evidence across modalities and pages

Enhances grounding and coherence in multimodal long-context QA

Abstract

Understanding multimodal long-context documents that comprise multimodal chunks such as paragraphs, figures, and tables is challenging due to (1) cross-modal heterogeneity to localize relevant information across modalities, (2) cross-page reasoning to aggregate dispersed evidence across pages. To address these challenges, we are motivated to adopt a query-centric formulation that projects cross-modal and cross-page information into a unified query representation space, with queries acting as abstract semantic surrogates for heterogeneous multimodal content. In this paper, we propose a Multimodal Long-Context Document Retrieval Augmented Generation (MLDocRAG) framework that leverages a Multimodal Chunk-Query Graph (MCQG) to organize multimodal document content around semantically rich, answerable queries. MCQG is constructed via a multimodal document expansion process that generates fine-grained queries from heterogeneous document chunks and links them to their corresponding content across modalities and pages. This graph-based structure enables selective, query-centric retrieval and structured evidence aggregation, thereby enhancing grounding and coherence in multimodal long-context question answering. Experiments on datasets MMLongBench-Doc and LongDocURL demonstrate that MLDocRAG consistently improves retrieval quality and answer accuracy, demonstrating its effectiveness for multimodal long-context understanding.