Decouple, Reorganize, and Fuse: A Multimodal Framework for Cancer Survival Prediction

TL;DR

This paper introduces DeReF, a multimodal framework for cancer survival prediction that enhances feature fusion and interaction through a novel reorganization strategy and regional cross-attention, improving predictive performance.

Contribution

The paper proposes a new Decoupling-Reorganization-Fusion framework with a random feature reorganization strategy and regional cross-attention to improve multimodal cancer survival prediction.

Findings

Effective on Liver Cancer and TCGA datasets

Outperforms existing fusion methods

Improves feature interaction and generalization

Abstract

Cancer survival analysis commonly integrates information across diverse medical modalities to make survival-time predictions. Existing methods primarily focus on extracting different decoupled features of modalities and performing fusion operations such as concatenation, attention, and \revm{Mixture-of-Experts (MoE)-based fusion. However, these methods still face two key challenges: i) Fixed fusion schemes (concatenation and attention) can lead to model over-reliance on predefined feature combinations, limiting the dynamic fusion of decoupled features; ii) in MoE-based fusion methods, each expert network handles separate decoupled features, which limits information interaction among the decoupled features. To address these challenges, we propose a novel Decoupling-Reorganization-Fusion framework (DeReF), which devises a random feature reorganization strategy between modalities decoupling and dynamic MoE fusion modules.Its advantages are: i) it increases the diversity of feature combinations and granularity, enhancing the generalization ability of the subsequent expert networks; ii) it overcomes the problem of information closure and helps expert networks better capture information among decoupled features. Additionally, we incorporate a regional cross-attention network within the modality decoupling module to improve the representation quality of decoupled features. Extensive experimental results on our in-house Liver Cancer (LC) and three widely used TCGA public datasets confirm the effectiveness of our proposed method. Codes are available at https://github.com/ZJUMAI/DeReF.