Few-shot Adaptation of Medical Vision-Language Models

TL;DR

This paper introduces the first structured benchmark for few-shot adaptation of medical vision-language models, evaluating simple and complex strategies across multiple modalities and tasks, revealing competitive performance of a linear-probe baseline.

Contribution

It presents a novel benchmark for few-shot medical VLM adaptation and demonstrates that a simple linear-probe method can be highly effective and efficient.

Findings

Linear-probe baseline performs competitively with complex methods.

The benchmark covers three medical modalities and nine tasks.

The code and benchmark are publicly available for further research.

Abstract

Integrating image and text data through multi-modal learning has emerged as a new approach in medical imaging research, following its successful deployment in computer vision. While considerable efforts have been dedicated to establishing medical foundation models and their zero-shot transfer to downstream tasks, the popular few-shot setting remains relatively unexplored. Following on from the currently strong emergence of this setting in computer vision, we introduce the first structured benchmark for adapting medical vision-language models (VLMs) in a strict few-shot regime and investigate various adaptation strategies commonly used in the context of natural images. Furthermore, we evaluate a simple generalization of the linear-probe adaptation baseline, which seeks an optimal blending of the visual prototypes and text embeddings via learnable class-wise multipliers. Surprisingly, such a text-informed linear probe yields competitive performances in comparison to convoluted prompt-learning and adapter-based strategies, while running considerably faster and accommodating the black-box setting. Our extensive experiments span three different medical modalities and specialized foundation models, nine downstream tasks, and several state-of-the-art few-shot adaptation methods. We made our benchmark and code publicly available to trigger further developments in this emergent subject: \url{https://github.com/FereshteShakeri/few-shot-MedVLMs}.