# Multimodal Deep Learning with Routine Clinical Data for Recurrence Risk Stratification in HR+/HER2− Early Breast Cancer

## TL;DR

A new AI tool combines clinical data and imaging to better predict recurrence risk in a common type of early breast cancer.

## Contribution

A multimodal deep learning model with cross-attention mechanisms for improved recurrence risk prediction in HR+/HER2− early breast cancer.

## Key findings

- The MRRP model achieved a C-index of 0.840, outperforming single-modality models.
- Pathology features were most critical, with ultrasound and clinical data adding complementary value.
- A learnable compensation mechanism improved robustness when data was missing.

## Abstract

Hormone receptor-positive/human epidermal growth factor receptor 2-negative (HR+/HER2−) early breast cancer (EBC) patients face long-term recurrence risk despite standard treatment. Current prognostic tools relying on clinicopathological factors or multigene assays have limited accuracy or accessibility. In this study, we developed a multimodal recurrence risk prediction (MRRP) model integrating routinely available clinical data, including whole-slide images (WISs), ultrasound (US) imaging and diagnostic reports, and structured clinical parameters. The MRRP model employs a hierarchical transformer-based fusion framework with innovative intra- and intermodality cross-attention mechanisms to dynamically integrate diverse feature representations. Using a well-curated cohort of 768 HR+/HER2− EBC patients with long-term follow-up, MRRP demonstrated superior prognostic performance (C-index = 0.840) compared to single-modality models, with robust time-dependent AUCs exceeding 0.85 at 3, 5, and 7 years. Ablation studies highlighted the central role of pathology features and the complementary value of US and clinical data. We further validated the optimal query selection strategies and evaluated different pretrained encoders, revealing complex modality interactions. To address real-world challenges of missing modality data, a learnable compensation mechanism was implemented, improving model robustness. Our study provides a clinically practical, AI-driven tool for precise risk stratification in HR+/HER2− EBC patients, facilitating individualized treatment and surveillance decisions without reliance on costly multi-omics data.