MG$^2$-RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation

TL;DR

MG$^2$-RAG introduces a hierarchical multimodal knowledge graph that enhances cross-modal reasoning and retrieval in large language models, achieving state-of-the-art results with reduced computational costs.

Contribution

It proposes a lightweight multi-granularity graph framework combining textual and visual information for improved multimodal reasoning and retrieval.

Findings

Achieves state-of-the-art performance on four multimodal tasks.

Reduces graph construction time by 43.3 times on average.

Cuts down costs by 23.9 times compared to existing graph-based methods.

Abstract

Retrieval-Augmented Generation (RAG) mitigates hallucinations in Multimodal Large Language Models (MLLMs), yet existing systems struggle with complex cross-modal reasoning. Flat vector retrieval often ignores structural dependencies, while current graph-based methods rely on costly ``translation-to-text'' pipelines that discard fine-grained visual information. To address these limitations, we propose \textbf{MG$^2$-RAG}, a lightweight \textbf{M}ulti-\textbf{G}ranularity \textbf{G}raph \textbf{RAG} framework that jointly improves graph construction, modality fusion, and cross-modal retrieval. MG$^2$-RAG constructs a hierarchical multimodal knowledge graph by combining lightweight textual parsing with entity-driven visual grounding, enabling textual entities and visual regions to be fused into unified multimodal nodes that preserve atomic evidence. Building on this representation, we introduce a multi-granularity graph retrieval mechanism that aggregates dense similarities and propagates relevance across the graph to support structured multi-hop reasoning. Extensive experiments across four representative multimodal tasks (i.e., retrieval, knowledge-based VQA, reasoning, and classification) demonstrate that MG$^2$-RAG consistently achieves state-of-the-art performance while reducing graph construction overhead with an average 43.3$\times$ speedup and 23.9$\times$ cost reduction compared with advanced graph-based frameworks.