Increasing Textual Context Size Boosts Medical Image-Text Matching

TL;DR

This paper introduces ClipMD, a simple method that enhances medical image-text matching by increasing textual context size, leading to state-of-the-art results in medical domains.

Contribution

The paper presents ClipMD, a novel approach that uses a sliding window technique to encode longer textual contexts for improved medical image-text matching.

Findings

ClipMD outperforms existing models on two medical datasets.

Increasing textual context size improves matching performance.

The approach is simple yet effective for medical image-text tasks.

Abstract

This short technical report demonstrates a simple technique that yields state of the art results in medical image-text matching tasks. We analyze the use of OpenAI's CLIP, a general image-text matching model, and observe that CLIP's limited textual input size has negative impact on downstream performance in the medical domain where encoding longer textual contexts is often required. We thus train and release ClipMD, which is trained with a simple sliding window technique to encode textual captions. ClipMD was tested on two medical image-text datasets and compared with other image-text matching models. The results show that ClipMD outperforms other models on both datasets by a large margin. We make our code and pretrained model publicly available.