# Foundation Model‐Enabled Multimodal Deep Learning for Prognostic Prediction in Colorectal Cancer with Incomplete Modalities: A Multi‐Institutional Retrospective Study

## TL;DR

FLARE is a new AI tool that combines multiple types of medical data to predict colorectal cancer outcomes, even when some data is missing, and it outperforms existing methods.

## Contribution

FLARE introduces a novel multimodal deep learning framework with foundation models and strategies for missing data in colorectal cancer prognosis.

## Key findings

- FLARE achieved the highest concordance index across all validation cohorts.
- The model effectively stratified patients into high- and low-risk groups.
- It demonstrated robust generalizability across four independent clinical centers.

## Abstract

Accurate prognostic prediction for colorectal cancer is essential for optimizing personalized treatment strategies and improving patient outcomes. Current unimodal approaches encounter significant limitations in effectively leveraging multimodal data and confront challenges with the issue of missing modalities. A novel multimodal deep learning framework named FLARE, which integrates pathological images, radiological imaging, and clinical text reports, is introduced to provide accurate risk assessments for colorectal cancer survival and progression. FLARE employs foundation models to achieve efficient feature extraction, utilizes an attention‐based multi‐branch framework to enhance synergy and distinctiveness across modalities, and incorporates a diversity‐promoting loss function. To address the issue of incomplete data, FLARE integrates modality and missing‐aware prompts, pseudo embeddings, and a modality‐level augmentation strategy, thereby effectively mitigating potential performance degradation. The performance of FLARE is retrospectively assessed using a dataset of 1679 colorectal cancer patients from four independent clinical centers. Its superior prognostic capability is demonstrated through Kaplan‐Meier analysis and the concordance index. FLARE effectively stratified patients into high‐ and low‐risk groups. It achieved the highest concordance index across all validation cohorts, significantly outperforming traditional clinical models and existing multimodal methods, thereby highlighting its robust generalizability. Interpretability was enhanced by the comprehensive analyses of clinical factors, immune infiltration patterns, and gene pathways, as well as visualizations of feature importance across multiple modalities. In summary, FLARE establishes a comprehensive and robust framework for multimodal deep learning in medical prognostics, providing an advanced Artificial intelligence, Multimodal Deep Learning, Prognosis prediction, colorectal cancer, foundation modeltool for precision cancer prognosis and intelligent diagnosis.

FLARE, a multimodal AI framework, combines pathology slides, radiology scans, and clinical reports to predict colorectal cancer outcomes, even when some tests are missing. Evaluated retrospectively in 1679 patients from four medical centers, it consistently achieved the best prognostic accuracy and clearly separated high‐ and low‐risk groups. Analyses of clinical factors and biological pathways add interpretability.