Multimodal Prototyping for cancer survival prediction

TL;DR

This paper introduces a multimodal cancer survival prediction framework that condenses histology images and transcriptomic data into prototypes, enabling efficient, interpretable, and accurate prognostication across multiple cancer types.

Contribution

The study proposes a novel prototype-based summarization method for multimodal data, reducing computational complexity and enhancing interpretability in survival prediction models.

Findings

Outperforms state-of-the-art methods on six cancer types

Achieves over 300x data compression without loss of accuracy

Enables new interpretability analyses of multimodal data

Abstract

Multimodal survival methods combining gigapixel histology whole-slide images (WSIs) and transcriptomic profiles are particularly promising for patient prognostication and stratification. Current approaches involve tokenizing the WSIs into smaller patches (>10,000 patches) and transcriptomics into gene groups, which are then integrated using a Transformer for predicting outcomes. However, this process generates many tokens, which leads to high memory requirements for computing attention and complicates post-hoc interpretability analyses. Instead, we hypothesize that we can: (1) effectively summarize the morphological content of a WSI by condensing its constituting tokens using morphological prototypes, achieving more than 300x compression; and (2) accurately characterize cellular functions by encoding the transcriptomic profile with biological pathway prototypes, all in an unsupervised fashion. The resulting multimodal tokens are then processed by a fusion network, either with a Transformer or an optimal transport cross-alignment, which now operates with a small and fixed number of tokens without approximations. Extensive evaluation on six cancer types shows that our framework outperforms state-of-the-art methods with much less computation while unlocking new interpretability analyses.