PhenoBench: A Comprehensive Benchmark for Cell Phenotyping

TL;DR

PhenoBench introduces a new comprehensive benchmark and dataset for cell phenotyping in histopathology images, revealing significant challenges for existing foundational models and providing a platform for future research.

Contribution

The paper presents PhenoCell, a new dataset with detailed cell types, and a benchmarking framework for evaluating foundational models on cell phenotyping tasks.

Findings

Existing models perform poorly on PhenoCell with scores as low as 0.20.

PhenoBench reveals greater challenges in cell phenotyping than previous benchmarks.

Benchmarking insights into model generalization under domain shifts.

Abstract

Digital pathology has seen the advent of a wealth of foundational models (FM), yet to date their performance on cell phenotyping has not been benchmarked in a unified manner. We therefore propose PhenoBench: A comprehensive benchmark for cell phenotyping on Hematoxylin and Eosin (H&E) stained histopathology images. We provide both PhenoCell, a new H&E dataset featuring 14 granular cell types identified by using multiplexed imaging, and ready-to-use fine-tuning and benchmarking code that allows the systematic evaluation of multiple prominent pathology FMs in terms of dense cell phenotype predictions in different generalization scenarios. We perform extensive benchmarking of existing FMs, providing insights into their generalization behavior under technical vs. medical domain shifts. Furthermore, while FMs achieve macro F1 scores > 0.70 on previously established benchmarks such as Lizard and PanNuke, on PhenoCell, we observe scores as low as 0.20. This indicates a much more challenging task not captured by previous benchmarks, establishing PhenoCell as a prime asset for future benchmarking of FMs and supervised models alike. Code and data are available on GitHub.