HeteroRAG: A Heterogeneous Retrieval-Augmented Generation Framework for Medical Vision Language Tasks

TL;DR

HeteroRAG is a novel framework that improves medical vision-language models by effectively retrieving and integrating heterogeneous knowledge sources, significantly enhancing factual accuracy and reliability in clinical tasks.

Contribution

The paper introduces HeteroRAG, a new retrieval-augmented generation framework that leverages modality-specific retrieval and multi-source knowledge alignment for medical vision-language tasks.

Findings

HeteroRAG achieves state-of-the-art results on 11 datasets.

Significant improvements in factual accuracy and reliability.

Effective retrieval across heterogeneous sources enhances clinical decision-making.

Abstract

Medical large vision-language Models (Med-LVLMs) have shown promise in clinical applications but suffer from factual inaccuracies and unreliable outputs, posing risks in real-world diagnostics. While RAG has emerged as a potential solution, current medical multimodal RAG systems are unable to perform effective retrieval across heterogeneous sources. The irrelevance of retrieved reports undermines the factuality of analysis, while insufficient knowledge affects the credibility of clinical decision-making. To bridge the research gap, we construct MedAtlas, which includes extensive multimodal report repositories and diverse text corpora. Based on it, we present HeteroRAG, a novel framework that enhances Med-LVLMs through heterogeneous knowledge sources. The framework introduces Modality-specific CLIPs for effective report retrieval and a Multi-corpora Query Generator for tailoring queries to diverse corpora. Incorporating knowledge from such multifaceted sources, Heterogeneous Knowledge Preference Tuning is performed to achieve cross-modality and multi-source knowledge alignment. Extensive experiments across 11 datasets and 3 modalities demonstrate that HeteroRAG achieves state-of-the-art performance in most medical vision language benchmarks, significantly improving factual accuracy and reliability of Med-LVLMs.