MMSF: Multitask and Multimodal Supervised Framework for WSI Classification and Survival Analysis

TL;DR

This paper introduces MMSF, a multitask and multimodal framework for whole slide image classification and survival analysis that effectively integrates heterogeneous pathological and clinical data, improving accuracy and prognostic performance.

Contribution

MMSF is a novel framework that explicitly decomposes and fuses multimodal signals using a graph-based approach and multitask learning, advancing multimodal computational pathology.

Findings

Achieved 2.1-6.6% accuracy improvements over baselines.

Improved AUC by 2.2-6.9% on classification tasks.

Enhanced C-index by 7.1-9.8% for survival analysis.

Abstract

Multimodal evidence is critical in computational pathology: gigapixel whole slide images capture tumor morphology, while patient-level clinical descriptors preserve complementary context for prognosis. Integrating such heterogeneous signals remains challenging because feature spaces exhibit distinct statistics and scales. We introduce MMSF, a multitask and multimodal supervised framework built on a linear-complexity MIL backbone that explicitly decomposes and fuses cross-modal information. MMSF comprises a graph feature extraction module embedding tissue topology at the patch level, a clinical data embedding module standardizing patient attributes, a feature fusion module aligning modality-shared and modality-specific representations, and a Mamba-based MIL encoder with multitask prediction heads. Experiments on CAMELYON16 and TCGA-NSCLC demonstrate 2.1--6.6\% accuracy and 2.2--6.9\% AUC improvements over competitive baselines, while evaluations on five TCGA survival cohorts yield 7.1--9.8\% C-index improvements compared with unimodal methods and 5.6--7.1\% over multimodal alternatives.