Feature Re-Embedding: Towards Foundation Model-Level Performance in Computational Pathology

TL;DR

This paper introduces R$^2$T, a re-embedding transformer module for MIL in computational pathology, enabling online feature fine-tuning and significantly improving model performance to foundation model levels.

Contribution

The proposed R$^2$T module allows online re-embedding of instance features, enhancing MIL model adaptability and performance without requiring pre-trained feature extractors.

Findings

Feature re-embedding boosts MIL performance to foundation model levels.

R$^2$T significantly improves various MIL models across tasks.

R$^2$T-MIL outperforms recent methods by a large margin.

Abstract

Multiple instance learning (MIL) is the most widely used framework in computational pathology, encompassing sub-typing, diagnosis, prognosis, and more. However, the existing MIL paradigm typically requires an offline instance feature extractor, such as a pre-trained ResNet or a foundation model. This approach lacks the capability for feature fine-tuning within the specific downstream tasks, limiting its adaptability and performance. To address this issue, we propose a Re-embedded Regional Transformer (R$^2$T) for re-embedding the instance features online, which captures fine-grained local features and establishes connections across different regions. Unlike existing works that focus on pre-training powerful feature extractor or designing sophisticated instance aggregator, R$^2$T is tailored to re-embed instance features online. It serves as a portable module that can seamlessly integrate into mainstream MIL models. Extensive experimental results on common computational pathology tasks validate that: 1) feature re-embedding improves the performance of MIL models based on ResNet-50 features to the level of foundation model features, and further enhances the performance of foundation model features; 2) the R$^2$T can introduce more significant performance improvements to various MIL models; 3) R$^2$T-MIL, as an R$^2$T-enhanced AB-MIL, outperforms other latest methods by a large margin.The code is available at: https://github.com/DearCaat/RRT-MIL.