Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Training

TL;DR

This paper introduces M$^3$AE, a self-supervised multi-modal masked autoencoder framework for medical vision-and-language pre-training, achieving state-of-the-art results on a new benchmark with innovative masking and reconstruction strategies.

Contribution

Proposes a novel multi-modal masked autoencoder approach for medical vision-and-language pre-training with tailored masking ratios and separate decoders, advancing the field's understanding and performance.

Findings

Achieves state-of-the-art results on medical vision-and-language tasks.

Demonstrates effectiveness of different masking ratios for images and texts.

Validates the importance of layer-specific feature reconstruction.

Abstract

Medical vision-and-language pre-training provides a feasible solution to extract effective vision-and-language representations from medical images and texts. However, few studies have been dedicated to this field to facilitate medical vision-and-language understanding. In this paper, we propose a self-supervised learning paradigm with multi-modal masked autoencoders (M$^3$AE), which learn cross-modal domain knowledge by reconstructing missing pixels and tokens from randomly masked images and texts. There are three key designs to make this simple approach work. First, considering the different information densities of vision and language, we adopt different masking ratios for the input image and text, where a considerably larger masking ratio is used for images. Second, we use visual and textual features from different layers to perform the reconstruction to deal with different levels of abstraction in visual and language. Third, we develop different designs for vision and language decoders (i.e., a Transformer for vision and a multi-layer perceptron for language). To perform a comprehensive evaluation and facilitate further research, we construct a medical vision-and-language benchmark including three tasks. Experimental results demonstrate the effectiveness of our approach, where state-of-the-art results are achieved on all downstream tasks. Besides, we conduct further analysis to better verify the effectiveness of different components of our approach and various settings of pre-training. The source code is available at~\url{https://github.com/zhjohnchan/M3AE}.