Going Beyond H&E and Oncology: How Do Histopathology Foundation Models Perform for Multi-stain IHC and Immunology?

TL;DR

This study assesses the transferability of histopathology foundation models to autoimmune IHC tasks, revealing limited improvements over ImageNet models and highlighting challenges in cross-domain generalization.

Contribution

It provides a comprehensive evaluation of 13 models on autoimmune IHC datasets, exposing limitations in current foundation models for diverse histopathological applications.

Findings

Histopathology-pretrained models did not outperform ImageNet-pretrained models.

Evidence of autoimmune feature misinterpretation and bias in model explanations.

Challenges in transferring knowledge from cancer to autoimmune histopathology.

Abstract

This study evaluates the generalisation capabilities of state-of-the-art histopathology foundation models on out-of-distribution multi-stain autoimmune Immunohistochemistry datasets. We compare 13 feature extractor models, including ImageNet-pretrained networks, and histopathology foundation models trained on both public and proprietary data, on Rheumatoid Arthritis subtyping and Sjogren's Disease detection tasks. Using a simple Attention-Based Multiple Instance Learning classifier, we assess the transferability of learned representations from cancer H&E images to autoimmune IHC images. Contrary to expectations, histopathology-pretrained models did not significantly outperform ImageNet-pretrained models. Furthermore, there was evidence of both autoimmune feature misinterpretation and biased feature importance. Our findings highlight the challenges in transferring knowledge from cancer to autoimmune histopathology and emphasise the need for careful evaluation of AI models across diverse histopathological tasks. The code to run this benchmark is available at https://github.com/AmayaGS/ImmunoHistoBench.