Rethinking Foundation Models for Medical Image Classification through a Benchmark Study on MedMNIST

TL;DR

This paper evaluates various foundation models, including convolutional and Transformer-based, on the MedMNIST dataset to assess their effectiveness in medical image classification, providing insights into model transferability and performance.

Contribution

It conducts a comprehensive benchmark study on foundation models for medical image classification, comparing end-to-end training and linear probing across multiple models and data settings.

Findings

Pre-trained models show strong potential for medical image classification.

Model performance varies with image size and training data amount.

Insights into transferability of different foundation models are provided.

Abstract

Foundation models are widely employed in medical image analysis, due to their high adaptability and generalizability for downstream tasks. With the increasing number of foundation models being released, model selection has become an important issue. In this work, we study the capabilities of foundation models in medical image classification tasks by conducting a benchmark study on the MedMNIST dataset. Specifically, we adopt various foundation models ranging from convolutional to Transformer-based models and implement both end-to-end training and linear probing for all classification tasks. The results demonstrate the significant potential of these pre-trained models when transferred for medical image classification. We further conduct experiments with different image sizes and various sizes of training data. By analyzing all the results, we provide preliminary, yet useful insights and conclusions on this topic.