Grounded Knowledge-Enhanced Medical Vision-Language Pre-training for Chest X-Ray

TL;DR

This paper introduces GK-MVLP, a grounded knowledge-enhanced pre-training framework for chest X-ray analysis that improves alignment between images and reports, leading to better performance across multiple medical vision-language tasks.

Contribution

The paper proposes a novel grounded knowledge-enhanced module that improves fine-grained alignment between visual features and medical knowledge in chest X-ray pre-training.

Findings

Outperforms state-of-the-art on disease classification and localization

Enhances report generation accuracy

Improves medical visual question-answering performance

Abstract

Medical foundation models have the potential to revolutionize healthcare by providing robust and generalized representations of medical data. Medical vision-language pre-training has emerged as a promising approach for learning domain-general representations of medical image and text. Current algorithms that exploit global and local alignment between medical image and text could however be marred by redundant information in medical data. To address this issue, we propose a grounded knowledge-enhanced medical vision-language pre-training (GK-MVLP) framework for chest X-ray. In this framework, medical knowledge was grounded to the appropriate anatomical regions by using a transformer-based grounded knowledge-enhanced module for fine-grained alignment between textural features of medical knowledge and the corresponding anatomical region-level visual features. The performance of GK-MVLP was competitive with or exceeded the state of the art on downstream image understanding tasks (chest X-ray disease classification, disease localization), generative task (report generation), and vision-language understanding task (medical visual question-answering). Our results demonstrate the advantage of incorporating grounding mechanism to remove biases and improve the alignment between chest X-ray image and radiology report.