Improving Medical Report Generation with Adapter Tuning and Knowledge Enhancement in Vision-Language Foundation Models

TL;DR

This paper enhances medical report generation by integrating adapter tuning and medical knowledge into vision-language models, significantly improving accuracy and coherence despite limited labeled data, validated on ImageCLEFmedical 2023.

Contribution

It introduces a novel approach combining adapter tuning and knowledge enhancement in vision-language models for medical report generation, addressing data scarcity challenges.

Findings

Achieved top results on ImageCLEFmedical 2023 dataset

Significant improvements in ROUGE and CIDEr scores

Demonstrated effective adaptation of foundation models to medical domain

Abstract

Medical report generation demands automatic creation of coherent and precise descriptions for medical images. However, the scarcity of labelled medical image-report pairs poses formidable challenges in developing large-scale neural networks capable of harnessing the potential of artificial intelligence, exemplified by large language models. This study builds upon the state-of-the-art vision-language pre-training and fine-tuning approach, BLIP-2, to customize general large-scale foundation models. Integrating adapter tuning and a medical knowledge enhancement loss, our model significantly improves accuracy and coherence. Validation on the dataset of ImageCLEFmedical 2023 demonstrates our model's prowess, achieving the best-averaged results against several state-of-the-art methods. Significant improvements in ROUGE and CIDEr underscore our method's efficacy, highlighting promising outcomes for the rapid medical-domain adaptation of the vision-language foundation models in addressing challenges posed by data scarcity.