Multi-task Paired Masking with Alignment Modeling for Medical Vision-Language Pre-training

TL;DR

This paper introduces a novel medical vision-language pre-training framework that enhances cross-modal alignment and interaction through multi-task paired masking, leading to improved performance across various downstream medical imaging tasks.

Contribution

It proposes a unified Med-VLP framework with multi-task paired masking, alignment modeling, and modules for better cross-modal interaction and semantic representation in medical imaging.

Findings

Outperforms previous methods in all downstream tasks

Enhances cross-modal interaction and semantic understanding

Improves performance in uni-, cross-, and multi-modal tasks

Abstract

In recent years, the growing demand for medical imaging diagnosis has placed a significant burden on radiologists. As a solution, Medical Vision-Language Pre-training (Med-VLP) methods have been proposed to learn universal representations from medical images and reports, benefiting downstream tasks without requiring fine-grained annotations. However, existing methods have overlooked the importance of cross-modal alignment in joint image-text reconstruction, resulting in insufficient cross-modal interaction. To address this limitation, we propose a unified Med-VLP framework based on Multi-task Paired Masking with Alignment (MPMA) to integrate the cross-modal alignment task into the joint image-text reconstruction framework to achieve more comprehensive cross-modal interaction, while a Global and Local Alignment (GLA) module is designed to assist self-supervised paradigm in obtaining semantic representations with rich domain knowledge. Furthermore, we introduce a Memory-Augmented Cross-Modal Fusion (MA-CMF) module to fully integrate visual information to assist report reconstruction and fuse the multi-modal representations adequately. Experimental results demonstrate that the proposed unified approach outperforms previous methods in all downstream tasks, including uni-modal, cross-modal, and multi-modal tasks.