Prompt-Guided Adaptive Model Transformation for Whole Slide Image Classification

TL;DR

This paper introduces PAMT, a novel framework that adapts pre-trained models for histopathology WSI classification by using prompt-guided transformations and domain-specific data reformulation, significantly improving performance.

Contribution

The paper proposes PAMT, a new prompt-guided adaptive model transformation method that effectively bridges the domain gap in WSI classification using MIL, with innovative data sampling and prompt techniques.

Findings

Substantial performance improvements on Camelyon16 and TCGA-NSCLC datasets.

PAMT outperforms existing MIL models in WSI classification accuracy.

Demonstrates the effectiveness of adaptive model transformation in domain adaptation.

Abstract

Multiple instance learning (MIL) has emerged as a popular method for classifying histopathology whole slide images (WSIs). Existing approaches typically rely on frozen pre-trained models to extract instance features, neglecting the substantial domain shift between pre-training natural and histopathological images. To address this issue, we propose PAMT, a novel Prompt-guided Adaptive Model Transformation framework that enhances MIL classification performance by seamlessly adapting pre-trained models to the specific characteristics of histopathology data. To capture the intricate histopathology distribution, we introduce Representative Patch Sampling (RPS) and Prototypical Visual Prompt (PVP) to reform the input data, building a compact while informative representation. Furthermore, to narrow the domain gap, we introduce Adaptive Model Transformation (AMT) that integrates adapter blocks within the feature extraction pipeline, enabling the pre-trained models to learn domain-specific features. We rigorously evaluate our approach on two publicly available datasets, Camelyon16 and TCGA-NSCLC, showcasing substantial improvements across various MIL models. Our findings affirm the potential of PAMT to set a new benchmark in WSI classification, underscoring the value of a targeted reprogramming approach.