MECFormer: Multi-task Whole Slide Image Classification with Expert Consultation Network

TL;DR

MECFormer is a Transformer-based multi-task model for whole slide image classification that integrates expert consultation and autoregressive decoding, achieving superior results across diverse datasets and tasks.

Contribution

The paper introduces MECFormer, a novel multi-task WSI classification model with an expert consultation network and autoregressive decoding, enabling flexible and effective multi-organ cancer diagnostics.

Findings

Outperforms state-of-the-art MIL models on five datasets

Effectively handles multiple tasks within a single model

Demonstrates versatility across different organs and cancer types

Abstract

Whole slide image (WSI) classification is a crucial problem for cancer diagnostics in clinics and hospitals. A WSI, acquired at gigapixel size, is commonly tiled into patches and processed by multiple-instance learning (MIL) models. Previous MIL-based models designed for this problem have only been evaluated on individual tasks for specific organs, and the ability to handle multiple tasks within a single model has not been investigated. In this study, we propose MECFormer, a generative Transformer-based model designed to handle multiple tasks within one model. To leverage the power of learning multiple tasks simultaneously and to enhance the model's effectiveness in focusing on each individual task, we introduce an Expert Consultation Network, a projection layer placed at the beginning of the Transformer-based model. Additionally, to enable flexible classification, autoregressive decoding is incorporated by a language decoder for WSI classification. Through extensive experiments on five datasets involving four different organs, one cancer classification task, and four cancer subtyping tasks, MECFormer demonstrates superior performance compared to individual state-of-the-art multiple-instance learning models.