CoRe-MMRAG: Cross-Source Knowledge Reconciliation for Multimodal RAG

TL;DR

This paper introduces CoRe-MMRAG, an end-to-end framework that reconciles inconsistencies across knowledge sources in multimodal retrieval-augmented generation, significantly improving performance on KB-VQA benchmarks.

Contribution

It proposes a novel four-stage pipeline and training paradigm to effectively address knowledge inconsistencies in multimodal RAG models.

Findings

Achieves 5.6% performance gain on InfoSeek

Achieves 9.3% performance gain on Encyclopedic-VQA

Outperforms baseline methods on KB-VQA benchmarks

Abstract

Multimodal Retrieval-Augmented Generation (MMRAG) has been introduced to enhance Multimodal Large Language Models by incorporating externally retrieved multimodal knowledge, but it introduces two challenges: Parametric-Retrieved Knowledge Inconsistency (PRKI), where discrepancies between parametric and retrieved knowledge create uncertainty in determining reliability, and Visual-Textual Knowledge Inconsistency (VTKI), where misalignment between visual and textual sources disrupts entity representation. To address these challenges, we propose Cross-source knowledge \textbf{Re}conciliation for Multimodal RAG (CoRe-MMRAG), a novel end-to-end framework that effectively reconciles inconsistencies across knowledge sources. CoRe-MMRAG follows a four-stage pipeline: it first generates an internal response from parametric knowledge, then selects the most relevant multimodal evidence via joint similarity assessment, generates an external response, and finally integrates both to produce a reliable answer. Additionally, a specialized training paradigm enhances knowledge source discrimination, multimodal integration, and unified answer generation. Experiments on KB-VQA benchmarks show that CoRe-MMRAG achieves substantial improvements over baseline methods, achieving 5.6% and 9.3% performance gains on InfoSeek and Encyclopedic-VQA, respectively.