FORESEE: Multimodal and Multi-view Representation Learning for Robust Prediction of Cancer Survival

TL;DR

FORESEE is an innovative multimodal framework that integrates pathology images and molecular data at multiple scales, employing cross-fusion transformers and attention mechanisms to improve cancer survival prediction robustness, even with missing data.

Contribution

The paper introduces FORESEE, a novel end-to-end multimodal learning framework combining cross-scale feature fusion and autoencoding to handle missing data in cancer survival prediction.

Findings

Outperforms state-of-the-art methods on four benchmark datasets.

Effectively handles intra-modality missing data.

Enhances feature representation through cross-fusion and attention mechanisms.

Abstract

Integrating the different data modalities of cancer patients can significantly improve the predictive performance of patient survival. However, most existing methods ignore the simultaneous utilization of rich semantic features at different scales in pathology images. When collecting multimodal data and extracting features, there is a likelihood of encountering intra-modality missing data, introducing noise into the multimodal data. To address these challenges, this paper proposes a new end-to-end framework, FORESEE, for robustly predicting patient survival by mining multimodal information. Specifically, the cross-fusion transformer effectively utilizes features at the cellular level, tissue level, and tumor heterogeneity level to correlate prognosis through a cross-scale feature cross-fusion method. This enhances the ability of pathological image feature representation. Secondly, the hybrid attention encoder (HAE) uses the denoising contextual attention module to obtain the contextual relationship features and local detail features of the molecular data. HAE's channel attention module obtains global features of molecular data. Furthermore, to address the issue of missing information within modalities, we propose an asymmetrically masked triplet masked autoencoder to reconstruct lost information within modalities. Extensive experiments demonstrate the superiority of our method over state-of-the-art methods on four benchmark datasets in both complete and missing settings.