Synergy vs. Noise: Performance-Guided Multimodal Fusion For Biochemical Recurrence-Free Survival in Prostate Cancer

TL;DR

This study investigates how multimodal deep learning models for prostate cancer prognosis perform when combining different data sources, emphasizing the importance of selecting high-quality modalities to avoid noise and improve predictive accuracy.

Contribution

We demonstrate that performance-guided modality selection is crucial in multimodal deep learning to enhance predictive accuracy in prostate cancer recurrence prediction.

Findings

High-performing modalities improve predictions when combined.

Weak modalities can degrade overall model performance.

Selective integration enhances multimodal model effectiveness.

Abstract

Multimodal deep learning (MDL) has emerged as a transformative approach in computational pathology. By integrating complementary information from multiple data sources, MDL models have demonstrated superior predictive performance across diverse clinical tasks compared to unimodal models. However, the assumption that combining modalities inherently improves performance remains largely unexamined. We hypothesise that multimodal gains depend critically on the predictive quality of individual modalities, and that integrating weak modalities may introduce noise rather than complementary information. We test this hypothesis on a prostate cancer dataset with histopathology, radiology, and clinical data to predict time-to-biochemical recurrence. Our results confirm that combining high-performing modalities yield superior performance compared to unimodal approaches. However, integrating a poor-performing modality with other higher-performing modalities degrades predictive accuracy. These findings demonstrate that multimodal benefit requires selective, performance-guided integration rather than indiscriminate modality combination, with implications for MDL design across computational pathology and medical imaging.