THUNDER: Tile-level Histopathology image UNDERstanding benchmark

TL;DR

THUNDER is a comprehensive benchmark for evaluating tile-level digital pathology models, focusing on performance, feature analysis, robustness, and uncertainty across diverse datasets and tasks.

Contribution

It introduces a dynamic, easy-to-use benchmark for comparing 23 foundation models on multiple datasets, emphasizing robustness and uncertainty in digital pathology.

Findings

23 models evaluated across 16 datasets

Insights into feature spaces and robustness

Benchmark supports diverse downstream tasks

Abstract

Progress in a research field can be hard to assess, in particular when many concurrent methods are proposed in a short period of time. This is the case in digital pathology, where many foundation models have been released recently to serve as feature extractors for tile-level images, being used in a variety of downstream tasks, both for tile- and slide-level problems. Benchmarking available methods then becomes paramount to get a clearer view of the research landscape. In particular, in critical domains such as healthcare, a benchmark should not only focus on evaluating downstream performance, but also provide insights about the main differences between methods, and importantly, further consider uncertainty and robustness to ensure a reliable usage of proposed models. For these reasons, we introduce THUNDER, a tile-level benchmark for digital pathology foundation models, allowing for efficient comparison of many models on diverse datasets with a series of downstream tasks, studying their feature spaces and assessing the robustness and uncertainty of predictions informed by their embeddings. THUNDER is a fast, easy-to-use, dynamic benchmark that can already support a large variety of state-of-the-art foundation, as well as local user-defined models for direct tile-based comparison. In this paper, we provide a comprehensive comparison of 23 foundation models on 16 different datasets covering diverse tasks, feature analysis, and robustness. The code for THUNDER is publicly available at https://github.com/MICS-Lab/thunder.