Single GPU Task Adaptation of Pathology Foundation Models for Whole Slide Image Analysis

TL;DR

This paper introduces TAPFM, a novel single-GPU method for adapting pathology foundation models to clinical tasks using weak labels, leveraging vision transformer attention for effective multiple instance learning.

Contribution

It proposes a stable, end-to-end adaptation approach for PFMs using vision transformer attention, enabling practical clinical application on standard hardware.

Findings

TAPFM outperforms conventional methods in mutation prediction tasks.

It effectively handles multi-label classification of actionable mutations.

The approach maintains stable training dynamics for end-to-end adaptation.

Abstract

Pathology foundation models (PFMs) have emerged as powerful tools for analyzing whole slide images (WSIs). However, adapting these pretrained PFMs for specific clinical tasks presents considerable challenges, primarily due to the availability of only weak (WSI-level) labels for gigapixel images, necessitating multiple instance learning (MIL) paradigm for effective WSI analysis. This paper proposes a novel approach for single-GPU \textbf{T}ask \textbf{A}daptation of \textbf{PFM}s (TAPFM) that uses vision transformer (\vit) attention for MIL aggregation while optimizing both for feature representations and attention weights. The proposed approach maintains separate computational graphs for MIL aggregator and the PFM to create stable training dynamics that align with downstream task objectives during end-to-end adaptation. Evaluated on mutation prediction tasks for bladder cancer and lung adenocarcinoma across institutional and TCGA cohorts, TAPFM consistently outperforms conventional approaches, with H-Optimus-0 (TAPFM) outperforming the benchmarks. TAPFM effectively handles multi-label classification of actionable mutations as well. Thus, TAPFM makes adaptation of powerful pre-trained PFMs practical on standard hardware for various clinical applications.