From Linear Probing to Joint-Weighted Token Hierarchy: A Foundation Model Bridging Global and Cellular Representations in Biomarker Detection

TL;DR

This paper introduces JWTH, a new foundation model that combines global and cell-level features for improved biomarker detection in pathology images, demonstrating significant accuracy gains across multiple tasks.

Contribution

JWTH is the first model to effectively integrate global patch embeddings with cell-centric information through self-supervised pretraining and attention pooling.

Findings

Up to 8.3% higher balanced accuracy

1.2% average improvement over prior models

Enhanced interpretability and robustness in biomarker detection

Abstract

AI-based biomarkers can infer molecular features directly from hematoxylin & eosin (H&E) slides, yet most pathology foundation models (PFMs) rely on global patch-level embeddings and overlook cell-level morphology. We present a PFM model, JWTH (Joint-Weighted Token Hierarchy), which integrates large-scale self-supervised pretraining with cell-centric post-tuning and attention pooling to fuse local and global tokens. Across four tasks involving four biomarkers and eight cohorts, JWTH achieves up to 8.3% higher balanced accuracy and 1.2% average improvement over prior PFMs, advancing interpretable and robust AI-based biomarker detection in digital pathology.