Medical Vision-Language Pre-Training for Brain Abnormalities

TL;DR

This paper introduces a domain-specific vision-language pretraining approach for brain abnormalities, utilizing automatically collected medical image-text data from public sources to enhance medical AI applications.

Contribution

It presents a novel pipeline for collecting and pretraining on medical image-text data, specifically for brain abnormalities, addressing domain-specific challenges.

Findings

Effective pretraining on medical data improves model understanding.

The pipeline facilitates large-scale data collection from public resources.

Model shows promising results in medical image-text tasks.

Abstract

Vision-language models have become increasingly powerful for tasks that require an understanding of both visual and linguistic elements, bridging the gap between these modalities. In the context of multimodal clinical AI, there is a growing need for models that possess domain-specific knowledge, as existing models often lack the expertise required for medical applications. In this paper, we take brain abnormalities as an example to demonstrate how to automatically collect medical image-text aligned data for pretraining from public resources such as PubMed. In particular, we present a pipeline that streamlines the pre-training process by initially collecting a large brain image-text dataset from case reports and published journals and subsequently constructing a high-performance vision-language model tailored to specific medical tasks. We also investigate the unique challenge of mapping subfigures to subcaptions in the medical domain. We evaluated the resulting model with quantitative and qualitative intrinsic evaluations. The resulting dataset and our code can be found here https://github.com/masoud-monajati/MedVL_pretraining_pipeline