MLIP: Medical Language-Image Pre-training with Masked Local Representation Learning

TL;DR

MLIP introduces a novel local relationship modeling and masked contrastive learning strategy to improve medical language-image pre-training, significantly enhancing data efficiency and performance in low-data scenarios.

Contribution

The paper proposes MLIP, a framework that leverages patch-sentence matching and semantic integrity estimation to better utilize limited medical image-text data.

Findings

Outperforms previous methods in zero-shot classification

Achieves superior results in few-shot segmentation

Enhances data efficiency in medical pre-training

Abstract

Existing contrastive language-image pre-training aims to learn a joint representation by matching abundant image-text pairs. However, the number of image-text pairs in medical datasets is usually orders of magnitude smaller than that in natural datasets. Besides, medical image-text pairs often involve numerous complex fine-grained correspondences. This paper aims to enhance the data efficiency by introducing multiple-to-multiple local relationship modeling to capture denser supervisions. More specifically, we propose a Medical Language-Image Pre-training (MLIP) framework, which exploits the limited image-text medical data more efficiently through patch-sentence matching. Furthermore, we introduce a masked contrastive learning strategy with semantic integrity estimation to reduce redundancy in images while preserving the underlying semantics. Our evaluation results show that MLIP outperforms previous work in zero/few-shot classification and few-shot segmentation tasks by a large margin.