Closing the gap in multimodal medical representation alignment

TL;DR

This paper investigates the modality gap in multimodal medical representation learning and proposes a framework to improve semantic alignment between medical images and text, enhancing retrieval and captioning tasks.

Contribution

It identifies the presence of the modality gap in medical multimodal data and introduces a modality-agnostic method to close this gap, improving semantic alignment.

Findings

Enhanced cross-modal retrieval accuracy

Improved medical image captioning quality

Reduced modality gap in medical data representations

Abstract

In multimodal learning, CLIP has emerged as the de-facto approach for mapping different modalities into a shared latent space by bringing semantically similar representations closer while pushing apart dissimilar ones. However, CLIP-based contrastive losses exhibit unintended behaviors that negatively impact true semantic alignment, leading to sparse and fragmented latent spaces. This phenomenon, known as the modality gap, has been partially mitigated for standard text and image pairs but remains unknown and unresolved in more complex multimodal settings, such as the medical domain. In this work, we study this phenomenon in the latter case, revealing that the modality gap is present also in medical alignment, and we propose a modality-agnostic framework that closes this gap, ensuring that semantically related representations are more aligned, regardless of their source modality. Our method enhances alignment between radiology images and clinical text, improving cross-modal retrieval and image captioning.