LLM-Guided Diagnostic Evidence Alignment for Medical Vision-Language Pretraining under Limited Pairing

TL;DR

This paper introduces LGDEA, a novel medical vision-language pretraining method that uses LLMs to align diagnostic evidence across modalities, reducing dependence on paired data and improving performance in medical image analysis tasks.

Contribution

The paper proposes evidence-level alignment guided by LLMs, enabling effective use of unpaired data and improving diagnostic representation learning in medical vision-language pretraining.

Findings

Significant improvements in phrase grounding, image-text retrieval, and zero-shot classification.

Rivals methods relying on large amounts of paired data.

Effective exploitation of unpaired medical images and reports.

Abstract

Most existing CLIP-style medical vision--language pretraining methods rely on global or local alignment with substantial paired data. However, global alignment is easily dominated by non-diagnostic information, while local alignment fails to integrate key diagnostic evidence. As a result, learning reliable diagnostic representations becomes difficult, which limits their applicability in medical scenarios with limited paired data. To address this issue, we propose an LLM-Guided Diagnostic Evidence Alignment method (LGDEA), which shifts the pretraining objective toward evidence-level alignment that is more consistent with the medical diagnostic process. Specifically, we leverage LLMs to extract key diagnostic evidence from radiology reports and construct a shared diagnostic evidence space, enabling evidence-aware cross-modal alignment and allowing LGDEA to effectively exploit abundant unpaired medical images and reports, thereby substantially alleviating the reliance on paired data. Extensive experimental results demonstrate that our method achieves consistent and significant improvements on phrase grounding, image--text retrieval, and zero-shot classification, and even rivals pretraining methods that rely on substantial paired data.