MIRA: A Novel Framework for Fusing Modalities in Medical RAG

TL;DR

MIRA is a new framework that improves factual accuracy in multimodal medical language models by dynamically managing retrieval and integrating external knowledge with internal reasoning.

Contribution

It introduces a novel retrieval adjustment mechanism and a medical RAG framework that together enhance factual correctness in medical AI applications.

Findings

Significantly improves factual accuracy in medical VQA and report generation.

Achieves state-of-the-art performance on medical benchmarks.

Demonstrates effective multimodal reasoning with external knowledge integration.

Abstract

Multimodal Large Language Models (MLLMs) have significantly advanced AI-assisted medical diagnosis, but they often generate factually inconsistent responses that deviate from established medical knowledge. Retrieval-Augmented Generation (RAG) enhances factual accuracy by integrating external sources, but it presents two key challenges. First, insufficient retrieval can miss critical information, whereas excessive retrieval can introduce irrelevant or misleading content, disrupting model output. Second, even when the model initially provides correct answers, over-reliance on retrieved data can lead to factual errors. To address these issues, we introduce the Multimodal Intelligent Retrieval and Augmentation (MIRA) framework, designed to optimize factual accuracy in MLLM. MIRA consists of two key components: (1) a calibrated Rethinking and Rearrangement module that dynamically adjusts the number of retrieved contexts to manage factual risk, and (2) A medical RAG framework integrating image embeddings and a medical knowledge base with a query-rewrite module for efficient multimodal reasoning. This enables the model to effectively integrate both its inherent knowledge and external references. Our evaluation of publicly available medical VQA and report generation benchmarks demonstrates that MIRA substantially enhances factual accuracy and overall performance, achieving new state-of-the-art results. Code is released at https://github.com/mbzuai-oryx/MIRA.