Is Self-Supervision Enough? Benchmarking Foundation Models Against End-to-End Training for Mitotic Figure Classification

TL;DR

This study benchmarks foundation models against end-to-end training for mitotic figure classification in histopathology, revealing that traditional end-to-end models outperform foundation models in accuracy and robustness.

Contribution

It provides a comprehensive comparison showing that foundation models do not outperform end-to-end training for this specific medical imaging task.

Findings

End-to-end-trained models outperform foundation models in accuracy.

Foundation models are not more robust to domain shifts.

Linear probing of foundation models does not match end-to-end training performance.

Abstract

Foundation models (FMs), i.e., models trained on a vast amount of typically unlabeled data, have become popular and available recently for the domain of histopathology. The key idea is to extract semantically rich vectors from any input patch, allowing for the use of simple subsequent classification networks potentially reducing the required amounts of labeled data, and increasing domain robustness. In this work, we investigate to which degree this also holds for mitotic figure classification. Utilizing two popular public mitotic figure datasets, we compared linear probing of five publicly available FMs against models trained on ImageNet and a simple ResNet50 end-to-end-trained baseline. We found that the end-to-end-trained baseline outperformed all FM-based classifiers, regardless of the amount of data provided. Additionally, we did not observe the FM-based classifiers to be more robust against domain shifts, rendering both of the above assumptions incorrect.