Bridging the Modality Bottleneck in Pathology MIL through Virtual Molecular Staining

TL;DR

The paper introduces MIST, a virtual molecular staining method that enhances pathology MIL models by incorporating molecular information during training without requiring transcriptomics at inference.

Contribution

MIST replaces the standard projection layer in MIL with a molecularly informed module trained on spatial transcriptomics, improving performance across multiple tasks.

Findings

MIST improves 240 of 256 configurations over standard projection layers.

Average gain of +3.5% across tasks, with +5.2% on survival prediction.

Gene-derived prototypes are key to the observed performance improvements.

Abstract

Multiple instance learning (MIL) is the dominant framework for whole-slide image analysis in computational pathology, typically combining a frozen patch encoder, a projection layer, and a slide-level aggregator. While encoders and aggregators have been extensively studied, the projection layer remains a largely morphology-only bottleneck. This limits endpoints such as biomarker status and survival, which are governed by a molecular state that is not fully captured by H&E morphology. We introduce Molecularly Informed Staining Transform (MIST), a plug-in replacement for the MIL projection layer that uses paired spatial transcriptomics only during training to construct virtual molecular stains. MIST clusters gene expression profiles into cross-modal prototypes, anchors them in the frozen foundation model feature space, and uses them to reorganize H&E patch features along molecularly guided axes. It requires no transcriptomics at inference and can be inserted before standard MIL aggregators. We evaluate MIST across 23 downstream tasks and 8 MIL aggregators. MIST improves 240 of 256 configurations over the standard projection layer, with an average gain of +3.5%, observed consistently across endpoint types: +5.2% on survival prediction, +3.3% on tissue subtyping, and +2.6% on biomarker prediction. Ablations confirm that gene-derived prototypes are the primary source of the gains, while spatial, biological, and pathological analyses show that cross-modal prototype affinities capture spatially coherent molecular programs from H&E alone.