When is a Foundation Model a Foundation Model

TL;DR

This paper evaluates the effectiveness of large foundation models in digital pathology retrieval tasks, finding they underperform compared to smaller, traditional deep networks, highlighting limitations in current applications.

Contribution

The study provides a critical assessment of foundation models in medical image-text retrieval, revealing their inferior performance in digital pathology relative to smaller models.

Findings

Foundation models perform worse than smaller networks in digital pathology retrieval.

Large models do not necessarily improve domain-specific tasks.

Current foundation models may have limitations in specialized medical applications.

Abstract

Recently, several studies have reported on the fine-tuning of foundation models for image-text modeling in the field of medicine, utilizing images from online data sources such as Twitter and PubMed. Foundation models are large, deep artificial neural networks capable of learning the context of a specific domain through training on exceptionally extensive datasets. Through validation, we have observed that the representations generated by such models exhibit inferior performance in retrieval tasks within digital pathology when compared to those generated by significantly smaller, conventional deep networks.