Can language-guided unsupervised adaptation improve medical image classification using unpaired images and texts?

TL;DR

This paper introduces MedUnA, a novel unsupervised adaptation method for vision-language models that leverages unpaired images and texts to improve medical image classification, especially when labeled data is scarce.

Contribution

MedUnA is the first approach to enable unsupervised learning of medical image classifiers using unpaired images and texts with vision-language models.

Findings

Significant accuracy improvements over zero-shot baselines.

Effective in scenarios with limited labeled medical images.

Applicable across multiple medical imaging datasets.

Abstract

In medical image classification, supervised learning is challenging due to the scarcity of labeled medical images. To address this, we leverage the visual-textual alignment within Vision-Language Models (VLMs) to enable unsupervised learning of a medical image classifier. In this work, we propose \underline{Med}ical \underline{Un}supervised \underline{A}daptation (\texttt{MedUnA}) of VLMs, where the LLM-generated descriptions for each class are encoded into text embeddings and matched with class labels via a cross-modal adapter. This adapter attaches to a visual encoder of \texttt{MedCLIP} and aligns the visual embeddings through unsupervised learning, driven by a contrastive entropy-based loss and prompt tuning. Thereby, improving performance in scenarios where textual information is more abundant than labeled images, particularly in the healthcare domain. Unlike traditional VLMs, \texttt{MedUnA} uses \textbf{unpaired images and text} for learning representations and enhances the potential of VLMs beyond traditional constraints. We evaluate the performance on three chest X-ray datasets and two multi-class datasets (diabetic retinopathy and skin lesions), showing significant accuracy gains over the zero-shot baseline. Our code is available at https://github.com/rumaima/meduna.