Enhancing Biomedical Multi-modal Representation Learning with Multi-scale Pre-training and Perturbed Report Discrimination

TL;DR

This paper introduces a novel pre-training method for biomedical vision-language models that uses perturbed report discrimination to better capture complex domain-specific semantics, leading to improved downstream task performance.

Contribution

It proposes a new contrastive learning approach with report perturbation and multi-granularity contrast, enhancing semantic understanding in biomedical multi-modal representations.

Findings

Outperforms baseline methods on multiple downstream tasks.

Learns more semantic and robust multi-modal representations.

Improves sensitivity to higher-level semantic structures.

Abstract

Vision-language models pre-trained on large scale of unlabeled biomedical images and associated reports learn generalizable semantic representations. These multi-modal representations can benefit various downstream tasks in the biomedical domain. Contrastive learning is widely used to pre-train vision-language models for general natural images and associated captions. Despite its popularity, we found biomedical texts have complex and domain-specific semantics that are often neglected by common contrastive methods. To address this issue, we propose a novel method, perturbed report discrimination, for pre-train biomedical vision-language models. First, we curate a set of text perturbation methods that keep the same words, but disrupt the semantic structure of the sentence. Next, we apply different types of perturbation to reports, and use the model to distinguish the original report from the perturbed ones given the associated image. Parallel to this, we enhance the sensitivity of our method to higher level of granularity for both modalities by contrasting attention-weighted image sub-regions and sub-words in the image-text pairs. We conduct extensive experiments on multiple downstream tasks, and our method outperforms strong baseline methods. The results demonstrate that our approach learns more semantic meaningful and robust multi-modal representations.