Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training

TL;DR

MedViLL is a novel BERT-based multi-modal model that effectively integrates radiology images and reports for diverse medical vision-language tasks, achieving superior performance across multiple datasets.

Contribution

The paper introduces MedViLL, a BERT-based architecture with a new multi-modal attention masking scheme tailored for medical image and report understanding and generation.

Findings

Outperforms baseline models on four downstream tasks

Demonstrates strong generalization across three datasets

Effective for both understanding and report generation tasks

Abstract

Recently a number of studies demonstrated impressive performance on diverse vision-language multi-modal tasks such as image captioning and visual question answering by extending the BERT architecture with multi-modal pre-training objectives. In this work we explore a broad set of multi-modal representation learning tasks in the medical domain, specifically using radiology images and the unstructured report. We propose Medical Vision Language Learner (MedViLL), which adopts a BERT-based architecture combined with a novel multi-modal attention masking scheme to maximize generalization performance for both vision-language understanding tasks (diagnosis classification, medical image-report retrieval, medical visual question answering) and vision-language generation task (radiology report generation). By statistically and rigorously evaluating the proposed model on four downstream tasks with three radiographic image-report datasets (MIMIC-CXR, Open-I, and VQA-RAD), we empirically demonstrate the superior downstream task performance of MedViLL against various baselines, including task-specific architectures. The source code is publicly available at: https://github.com/SuperSupermoon/MedViLL