Attention-Based Multimodal Survival Prediction with Cross-Modal Bilinear Fusion

TL;DR

This paper introduces a multimodal deep learning framework combining histology, RNA-seq, and clinical data for patient survival prediction, emphasizing interpretability and efficiency.

Contribution

It presents a novel fusion architecture using cross-modal bilinear interactions for improved multimodal survival prediction.

Findings

Outperforms concatenation-based baselines in predictive accuracy.

Demonstrates competitive generalization on unseen cohorts.

Provides a structurally interpretable and parameter-efficient fusion method.

Abstract

We propose a novel multimodal deep learning framework for patient-level survival prediction, which integrates whole-slide histology features, RNA-seq expression profiles, and clinical variables. Our architecture combines an ABMIL module~\cite{ilse2018attention} for slide-level representation with feedforward encoders for RNA and clinical data. These embeddings are then integrated through low-rank bilinear cross-modal fusion~\cite{liu2018efficient} to model conditional interactions across modalities while controlling parameter growth. The model outputs continuous risk scores that are subsequently mapped to survival times using a nonparametric calibration procedure based on the Kaplan--Meier estimator~\cite{kaplan1958nonparametric}. By decomposing multimodal reasoning into independent pairwise interactions, the proposed fusion design promotes structural interpretability and parameter efficiency compared with full tensor and hierarchical fusion strategies. Experiments on the CHIMERA challenge dataset demonstrate improved predictive performance over concatenation-based baselines and competitive generalization on hidden evaluation cohorts. These results indicate that the proposed framework is a promising approach for multimodal survival prediction in HR-NMIBC. The implementation is publicly available at https://github.com/hassancpu/ChimeraChallenge2025_Task_3.