Navigating Data Scarcity using Foundation Models: A Benchmark of Few-Shot and Zero-Shot Learning Approaches in Medical Imaging

TL;DR

This study benchmarks 16 foundation models on 19 medical imaging datasets to evaluate their effectiveness in few-shot and zero-shot learning scenarios, revealing that models pretrained on medical data excel with very limited samples.

Contribution

It provides a comprehensive comparison of foundation models for medical image analysis in low-data regimes, highlighting the best performers and identifying gaps for future research.

Findings

BiomedCLIP performs best with very small datasets.

Large CLIP models excel with more training samples.

Fine-tuning ResNet-18 on ImageNet is competitive with more than five samples per class.

Abstract

Data scarcity is a major limiting factor for applying modern machine learning techniques to clinical tasks. Although sufficient data exists for some well-studied medical tasks, there remains a long tail of clinically relevant tasks with poor data availability. Recently, numerous foundation models have demonstrated high suitability for few-shot learning (FSL) and zero-shot learning (ZSL), potentially making them more accessible to practitioners. However, it remains unclear which foundation model performs best on FSL medical image analysis tasks and what the optimal methods are for learning from limited data. We conducted a comprehensive benchmark study of ZSL and FSL using 16 pretrained foundation models on 19 diverse medical imaging datasets. Our results indicate that BiomedCLIP, a model pretrained exclusively on medical data, performs best on average for very small training set sizes, while very large CLIP models pretrained on LAION-2B perform best with slightly more training samples. However, simply fine-tuning a ResNet-18 pretrained on ImageNet performs similarly with more than five training examples per class. Our findings also highlight the need for further research on foundation models specifically tailored for medical applications and the collection of more datasets to train these models.