Integrating Pathology and CT Imaging for Personalized Recurrence Risk Prediction in Renal Cancer

TL;DR

This study demonstrates that integrating preoperative CT scans with postoperative pathology images using deep learning improves personalized recurrence risk prediction in renal cancer, outperforming traditional clinical scores.

Contribution

It introduces a modular deep learning framework for multimodal data integration that enhances recurrence risk prediction accuracy in ccRCC.

Findings

WSI-based models outperform CT-only models

Intermediate fusion improves prediction performance

The best model approaches the performance of the Leibovich score

Abstract

Recurrence risk estimation in clear cell renal cell carcinoma (ccRCC) is essential for guiding postoperative surveillance and treatment. The Leibovich score remains widely used for stratifying distant recurrence risk but offers limited patient-level resolution and excludes imaging information. This study evaluates multimodal recurrence prediction by integrating preoperative computed tomography (CT) and postoperative histopathology whole-slide images (WSIs). A modular deep learning framework with pretrained encoders and Cox-based survival modeling was tested across unimodal, late fusion, and intermediate fusion setups. In a real-world ccRCC cohort, WSI-based models consistently outperformed CT-only models, underscoring the prognostic strength of pathology. Intermediate fusion further improved performance, with the best model (TITAN-CONCH with ResNet-18) approaching the adjusted Leibovich score. Random tie-breaking narrowed the gap between the clinical baseline and learned models, suggesting discretization may overstate individualized performance. Using simple embedding concatenation, radiology added value primarily through fusion. These findings demonstrate the feasibility of foundation model-based multimodal integration for personalized ccRCC risk prediction. Future work should explore more expressive fusion strategies, larger multimodal datasets, and general-purpose CT encoders to better match pathology modeling capacity.

Tables

Table 1. Table 1: C-index and AUROC for unimodal models, multimodal fusion strategies, and clinical baseline. Best learned results are in bold , second-best are underlined . Leibovich shown in bold † to denote highest overall performance; Leibovich (RT) uses randomized tie-breaking to adjust for score discretization.

Strategy	WSI Encoder	CT Encoder	C-index $\uparrow$	AUROC $\uparrow$
Unimodal	TITAN–CONCH	–	0.745±0.046	0.771±0.081
	CHIEF–CTransPath	–	0.663±0.054	0.601±0.152
	–	ResNet-10	0.547±0.119	0.623±0.141
	–	ResNet-18	0.518±0.124	0.532±0.200
	–	SwinUNETR	0.549±0.129	0.604±0.154
Late	TITAN–CONCH	ResNet-10	0.727±0.106	0.763±0.088
		ResNet-18	0.732±0.048	0.741±0.088
		SwinUNETR	0.739±0.069	0.748±0.110
	CHIEF–CTransPath	ResNet-10	0.692±0.068	0.597±0.162
		ResNet-18	0.666±0.046	0.627±0.190
		SwinUNETR	0.692±0.029	0.606±0.153
Intermediate	TITAN–CONCH	ResNet-10	0.742±0.086	0.780±0.133
		ResNet-18	0.775±0.044	0.774±0.115
		SwinUNETR	0.749±0.083	0.768±0.063
	CHIEF–CTransPath	ResNet-10	0.691±0.122	0.671±0.093
		ResNet-18	0.669±0.112	0.649±0.170
		SwinUNETR	0.679±0.068	0.624±0.201
Leibovich	–	–	0.805±0.035†	0.814±0.141†
Leibovich (RT)	–	–	0.749±0.043	0.814±0.141†

Equations

$$h(t\mid {\mathbf{x}}_i)=h_0(t)\exp (f({\mathbf{x}}_i))$$

$${\mathcal{L}}_{\text{Cox}}=-\sum _{i:{\delta }_i=1}\left(f({\mathbf{x}}_i)-\log \sum _{j\in \mathcal{R}(T_i)}\exp (f({\mathbf{x}}_j))\right),$$

$$\text{C-index}=\frac{{\sum }_{i,j}{1}_{\{T_j<T_i\}}\cdot 1_{\{{\eta }_j>{\eta }_i\}}\cdot {\delta }_j}{{\sum }_{i,j}{1}_{\{T_j<T_i\}}\cdot {\delta }_j}$$

$${\text{AUROC}}_{t=5}=\frac{{\sum }_{i,j}[Y_i(5)=1\wedge Y_j(5)=0\wedge {\eta }_i>{\eta }_j]}{{\sum }_{i,j}[Y_i(5)=1\wedge Y_j(5)=0]}$$

Full text

11footnotetext: These authors contributed equally to this work.

1 Introduction

Clear cell renal cell carcinoma (ccRCC) is the most common subtype of renal cancer and carries a substantial risk of recurrence after surgery, even when radical nephrectomy is performed with curative intent [5]. Estimating recurrence risk is important for determining patient-specific follow-up schedules and the potential use of adjuvant therapies. In clinical practice, the Leibovich score is widely used to stratify the risk of distant recurrence based on a limited set of histopathologic features, including tumor stage, size, grade, and presence of necrosis [17, 18]. Although clinically validated and easy to apply, this score provides only a population-level risk estimate, relies on pathologist evaluations that can be labor-intensive, and may lack the resolution needed for personalized decision-making.

Previous studies have proposed data-driven approaches to improve risk prediction. Margue et al. [20] showed that structured clinical and pathology report data can be used to expand the predictive feature set beyond that of the Leibovich score, allowing for more individualized risk estimation. Gui et al. [10] combined clinical, genomic, and pathological whole-slide images (WSIs) in a multimodal framework and found that this integration improved performance, with WSI-derived features contributing most strongly to the model. These findings support the idea that integrating complementary data sources can enhance recurrence prediction. However, to our knowledge, no prior study has systematically combined WSI and radiological imaging for this task, despite their widespread availability and potential complementarity.

Recent advances in foundation models trained on large-scale medical imaging datasets have made it increasingly feasible to integrate multiple imaging modalities. These models provide versatile feature representations that can be transferred to new tasks, enabling effective multimodal integration even when training data is limited. This study investigates the use of both postoperative WSI and preoperative computed tomography (CT) imaging for distant recurrence risk prediction in ccRCC. A modular deep learning pipeline is developed using feature representations derived from foundation models trained on large-scale medical imaging datasets. Both unimodal and multimodal settings are explored, incorporating intermediate and late fusion techniques. Model performance is benchmarked against the clinical Leibovich score to evaluate whether imaging-based integration can provide more individualized recurrence risk estimates. The approach is implemented on a publicly available dataset, enabling a systematic exploration of imaging modality combinations in a realistic clinical context. This work aims to support the development of clinically meaningful, data-driven decision support tools by integrating multimodal data to enable more personalised follow-up strategies. The main contributions of this work are as follows:

1. 1.

A modular multimodal modeling framework that leverages foundation models to extract meaningful feature representations from WSI and CT data, enabling flexible experimentation with both late and intermediate fusion strategies under limited supervision. 2. 2.

Interpretability analysis of modality-specific and fused representations for recurrence risk prediction in ccRCC, including performance comparisons against unimodal baselines and the clinically established Leibovich score. 3. 3.

Benchmarking on a publicly available, real-world cohort to provide a reproducible evaluation of fusion strategies and assess the complementary prognostic value of anatomical and histopathological imaging in a realistic clinical setting.

2 Methodology

Figure 1 provides an overview of the proposed modeling pipeline of this study. The following sections describe the details of feature representation, fusion strategies, and outcome modeling.

Cross-Modality Feature Representation. WSI representations are derived using the Trident framework [25, 22], which performs tissue segmentation, patch tiling, and slide-level aggregation. Two encoder combinations are employed: the patch encoder CONCHv1.5 [19] with the slide encoder TITAN [9], and the patch encoder CTransPath [23] with the slide encoder CHIEF [24]. In both cases, patch embeddings are aggregated into a single patient-level vector by the corresponding slide encoder. All pathology encoders are kept frozen during training due to their scale and pretraining complexity. CT features are extracted using pretrained 3D encoders applied to resampled, cropped, and intensity-normalized volumes. Two architectures are used: MedicalNet [6], a 3D ResNet pretrained on diverse CT and MRI datasets, and SwinUNETR [13], a hybrid transformer with U-Net–style decoding pretrained on BraTS brain MRI data [3]. In both cases, the segmentation head is removed, and the final encoder block is used to generate a volume-level feature vector. These encoders are fine-tuned for recurrence prediction: for MedicalNet, only the final residual block is updated; for SwinUNETR, only the CNN encoder component is trained. Fine-tuning is feasible due to the smaller model size relative to pathology encoders. Each CT scan is represented by a fixed-length vector, obtained via a fully connected layer that maps the encoding to the WSI embedding dimension, ensuring balanced representation across modalities.

Unified Prediction Framework. All models use a Cox-based multilayer perceptron (MLP) prediction architecture that maps input features to a scalar risk score. The MLP consists of an input layer, one hidden layer with ReLU activation and layer normalization, and an output layer producing the final prediction. While input dimensionality varies by modality or fusion strategy, the overall structure stays the same. Each model variant is trained separately using the Cox partial likelihood loss, with independent tuning of the randomly initialized prediction head to fit modality-specific representations. In the unimodal setting, patient-level feature vectors from either CT or WSI data are passed independently through modality-specific MLPs. These models are trained separately and serve as baselines to assess the predictive value of each imaging modality alone. In the late fusion approach, CT and WSI models are first trained independently. Their output risk scores $R_{\textbf{i}}$ are then combined post hoc using a weighted average: $R=\alpha\cdot R_{\text{WSI}}+(1-\alpha)\cdot R_{\text{CT}}$, where $\alpha\in[0,1]$ controls the relative contribution of each modality. In the intermediate fusion strategy, CT and WSI feature vectors are concatenated into a joint representation, which is passed through a single MLP. This setup enables the model to learn feature interactions between modalities and capture complementary information during training.

Survival Outcome Modeling via Cox Partial Likelihood Estimation. To estimate recurrence risk from imaging-derived features, all models are trained using a Cox proportional hazards framework [7]. This approach has also been adopted in deep learning-based survival models, such as DeepSurv [16], which optimize the Cox partial likelihood using neural networks. Given a set of patients indexed by $i=1,\dots,N$, each with a feature vector $\mathbf{x}_{i}$, the model outputs a scalar risk score $f(\mathbf{x}_{i})\in\mathbb{R}$. A patient’s hazard function $h(t\mid\mathbf{x}_{i})$ is defined as:

[TABLE]

where $h_{0}(t)$ is an unspecified baseline hazard function, and $\exp(f(\mathbf{x}_{i}))$ represents the relative risk associated with the patient’s covariates. This formulation captures the ordering of recurrence events without requiring precise estimation of survival times. In clinical datasets, not all patients experience recurrence within the observation window, making it essential to account for censoring when learning from event times. Model training is therefore based on a Cox partial likelihood loss objective, which for uncensored recurrence times $T_{i}$ is given by:

[TABLE]

where $\delta_{i}\in\{0,1\}$ is the event indicator, and $\mathcal{R}(T_{i})$ denotes the risk set, which includes all patients still at risk at time $T_{i}$. This loss function encourages the model to assign higher risk scores to patients who experience earlier recurrence, thereby preserving a concordant ordering with observed outcomes.

3 Experiments

Datasets. The Cancer Genome Atlas Kidney Renal Clear Cell Carcinoma Collection (TCGA-KIRC) dataset [21, 2] includes contrast-enhanced (CE)-CT scans, resected WSIs, and clinical annotations with postoperative distant recurrence outcomes, making it suitable for ccRCC recurrence evaluation. From the full cohort, 187 patients with both CT and WSI data were identified. After manual review, 156 patients were retained based on the availability of abdominal CTs with sufficient image quality, appropriate contrast phase (late arterial or early venous), and visible kidney structures. The final cohort had complete imaging and survival data suitable for time-to-event modeling. Patients had a median age of 57.6 years (range 26–81), with 70.5% male. Recurrence occurred in 40 patients (25.6%), while 116 (74.4%) remained disease-free during follow-up. Tumor staging was mainly T1 (55.2%) and T3 (30.1%), with rare nodal involvement (1.9% N1) and undetermined nodal status in 83 patients (53.2%). Histologic grades were mostly 2 (42.3%) and 3 (46.8%).

Experimental Protocol. Experiments were conducted using nested stratified cross-validation with five outer folds and three inner folds. Stratification was based on recurrence status, and fold assignments were fixed across all experiments to ensure consistent evaluation across model types. To focus the model on anatomically relevant regions, 3D CT volumes were first resampled to a uniform voxel spacing based on the average resolution across the dataset. Kidney masks were then generated using a nnU-NetV2 model [15] trained on the KiTS21 dataset [14]. The largest kidney volume in the training set was used to define a fixed bounding box of size $392\times 256\times 126$, which was consistently applied to all resampled images by cropping around the kidneys. Volumes were subsequently downsampled to $196\times 128\times 8$ for MedicalNet [6] and $196\times 128\times 32$ for SwinUNETR [13] to match the input requirements of the respective pretrained encoders. This process standardized spatial resolution, focused attention on the kidney region, and reduced computational cost while preserving relevant anatomy. Diagnostic-quality WSIs from the TCGA-KIRC dataset were selected based on expert-rated slide annotations. Preprocessing was performed using the Trident framework with default parameters. All slides were processed at the default magnification level of 20, ensuring consistent spatial resolution across patients and compatibility with the pretrained encoders.

Training was conducted using the Adam optimizer with a cosine annealing scheduler and linear warmup. In the inner cross-validation loop, models were trained for up to 200 epochs with early stopping based on validation concordance index (C-index), using a patience of 10 epochs. The optimal number of epochs was selected per outer fold based on inner-loop performance and used to train the final model on the corresponding outer training set. Bayesian hyperparameter optimization was performed using Optuna [1], targeting the average validation C-index across inner folds. Tuned hyperparameters and their respective search ranges included: learning rate $[10^{-5},10^{-3}]$, weight decay $[10^{-6},10^{-2}]$, L1 penalty $[10^{-6},10^{-2}]$, hidden layer size $[32,512]$, dropout rate $[0.0,0.5]$, and cosine scheduler floor $[10^{-6},10^{-2}]$. For late fusion models, the fusion weight $\alpha$, controlling the contribution of each modality, was also tuned over the range $[0,1]$. All models were trained with a batch size of 16, and final evaluation was performed using the best inner-fold configuration for each outer fold.

Adjusted Leibovich score as a Clinical Baseline for Risk Stratification. The Leibovich score is a clinically validated tool for estimating the risk of non-local recurrence in ccRCC following nephrectomy. It integrates pathological features including tumor size, nuclear grade, T-stage, lymph node involvement, and tumor necrosis [17, 18], and remains widely used in clinical practice [4]. In this study, the score was computed from structured pathology and surgical data. Necrosis was omitted due to its absence in the dataset, resulting in an adjusted score. Accordingly, standard risk group thresholds (i.e., low, intermediate, and high) were not applied, as they may not be valid without the necrosis component. The adjusted score ranged from 0 to 9, with 41.0% of patients scoring below 3 and 91.0% scoring below 6. This discrete score was used as a baseline risk estimate and evaluated using the same time-to-event metrics as the learned models, enabling consistent comparison of recurrence risk discrimination.

Evaluation Metrics. The primary evaluation metric is the C-index [12], a generalization of the area under the receiver operating characteristic curve (AUROC) [11] that accounts for right-censored data. It quantifies the model’s ability to accurately rank patients by recurrence risk. The C-index is defined as:

[TABLE]

where $T_{i}$ is the observed time for patient $i$, $\delta_{i}\in\{0,1\}$ is the event indicator, and $\eta_{i}=f(\mathbf{x}_{i})$ is the predicted risk score. Tied predictions are assigned a concordance of 0.5, following the implementation in the Python package lifelines [8]. A C-index of 1.0 indicates perfect ranking, while 0.5 indicates random performance. The AUROC for 5-year disease-free survival (DFS) is reported as a secondary metric to align with prior studies. Patients who experience recurrence within 5 years are labeled positive; those who remain recurrence-free beyond 5 years, including censored patients, are labeled negative. The AUROC evaluates the model’s ability to separate these two groups:

[TABLE]

where $Y_{i}(5)\in\{0,1\}$ is the binary indicator of whether patient $i$ experienced recurrence within five years.

4 Results

Predictive Model Performance. Table 1 summarizes predictive performance across unimodal models, fusion strategies, and the clinical baseline using five-fold cross-validation and described evaluation metrics. The Leibovich score achieved the highest overall performance (C-index $0.8046\pm 0.0348$; AUROC $0.8139\pm 0.1412$), consistent with its established role in postoperative risk stratification. However, its discretized nature results in many patients receiving identical scores, which limits granularity and potentially inflates the C-index under standard tie-handling. When ties were instead resolved randomly, closer to how such scores would be used in practice, the mean C-index dropped to $0.7494\pm 0.0430$, more closely aligning with the top-performing learned models. This demonstrates that, despite its overall clinical effectiveness, the discretized nature of the Leibovich score may limit its utility in personalized decision support.

Performance Analysis of Unimodal and Multimodal Strategies. Among learned unimodal models, those based on histopathology consistently outperformed radiology-based models. This is not unexpected, as many established prognostic features are histologic in nature. In contrast, CT-derived models showed limited predictive value, possibly due to domain mismatch in pretraining, which fine-tuning may not fully overcome given limited data. Most radiology encoders were originally developed for segmentation tasks, which may limit their effectiveness for survival prediction compared to pathology foundation models, which are trained for general-purpose feature extraction across tasks.

Fusion strategies, however, revealed that CT can still provide complementary signal when combined with pathology. In the multimodal late fusion setting, fusion weights across models consistently assigned high importance to WSI-derived predictions, with $\alpha$-values ranging from 0.8 to 0.9, indicating that pathology contributed the majority of the predictive signal. Importantly, intermediate fusion consistently outperformed both unimodal and late fusion models, with all WSI–CT combinations matching or exceeding their WSI-only baselines. The best result was achieved by the combination of TITAN–CONCH and the ResNet-18 model (C-index $0.775\pm 0.044$; AUROC $0.774\pm 0.115$).

Models using the TITAN–CONCH pathology encoder consistently achieved the strongest performance, suggesting this combination is particularly effective at capturing histologic risk patterns relevant to recurrence. Among CT encoders, ResNet-18 outperformed both the shallower ResNet-10 and the transformer-based SwinUNETR in intermediate fusion, despite being the lowest-performing model in the unimodal CT setting. This discrepancy highlights substantial variability in CT model performance and suggests that certain encoders may only reveal their predictive value when combined with complementary modalities. The relatively high standard deviations observed across CT-based models further underscore the instability of radiologic representations in this setting.

While some intermediate fusion models approached the clinical baseline and outperformed their unimodal and late fusion counterparts, these gains should be interpreted with caution due to their reported standard deviations. All fusion models used a simple concatenation-based strategy, suggesting that more expressive techniques such as cross-attention or co-learning architectures may further enhance integration and performance. Although pretrained encoders helped mitigate data scarcity, particularly for the information-rich pathology modality, the radiology side may benefit from the development of general-purpose foundation models trained across diverse tasks and anatomical regions. This could enhance the robustness of radiologic representations and align them more closely with the modeling capabilities observed in pathology. Future work could also explore more advanced fusion mechanisms, richer multimodal datasets, and the incorporation of additional data types to further enhance personalized recurrence risk prediction.

Case-based Analysis of Prediction Discrepancies.

A preliminary review of three representative cases was conducted to illustrate different patterns of concordance between the multimodal model and the Leibovich score: one where the clinical score performed better, one showing agreement, and one where the model outperformed the clinical score (Fig. 2). Predictions are shown for the best-performing model: intermediate fusion of TITAN–CONCH and ResNet-18. In Case A, the model substantially underestimated recurrence risk, despite an event occurring within 9 months. The selected WSI lacked overt high-risk features, although the pathology report noted necrosis and high Fuhrman grade, suggesting the slide may not have captured the most aggressive tumor regions. While the CT scan showed extensive disease, the radiology encodings may not have conveyed this effectively, highlighting potential limitations in both spatial sampling and feature representation. Case B showed strong concordance, with both model and score predicting high risk, consistent with early recurrence. Apparent perinephric fat invasion was observed, making this a relatively straightforward case where classical pathologic and radiologic features aligned well with the clinical outcome. In contrast, Case C illustrates a scenario where the model predicted high risk not reflected in the Leibovich score. This discrepancy may reflect the model’s ability to capture subtle histologic or radiologic signals not explicitly encoded in structured clinical variables, underscoring the potential of learned, data-driven representations to complement existing tools.

5 Conclusion

This study investigated postoperative distant recurrence risk prediction in ccRCC using a multimodal imaging framework that combines preoperative CT and postoperative WSI data. A modular pipeline with pretrained encoders and Cox-based survival models was evaluated in unimodal and multimodal late and intermediate fusion settings, with an adjusted Leibovich score as a benchmark. WSI-based models consistently outperformed CT-only models, reflecting the substantial prognostic value of pathology. Intermediate fusion further improved performance, with the best learned model, TITAN–CONCH combined with ResNet-18, approaching that of the adjusted Leibovich score. This suggests that CT and WSI offer complementary prognostic information that, when integrated, can enhance risk estimation. The discretized nature of the Leibovich score may overstate its performance for individualized prediction, as random tie-breaking brought its concordance closer to the learned models. These results were achieved using basic embedding concatenation, with strong fusion performance even from weaker unimodal CT encoders such as ResNet-18. This highlights the untapped potential in radiologic data when effectively integrated. Dataset size likely limited model performance, particularly for CT, and contributed to encoder variability. Future work should explore more expressive fusion strategies, such as cross-attention or co-learning, and develop general-purpose CT foundation models analogous to those in pathology. Scaling to larger multimodal datasets and improving interpretability will be key for enabling clinically trustworthy, personalized decision support systems.

5.0.1 Acknowledgements.

Research at the Netherlands Cancer Institute is supported by the Dutch Cancer Society and the Dutch Ministry of Health, Welfare and Sport. This publication is part of the project ”Ordinality-informed Federated Learning for Robust and Explainable Radiology AI” with file number NGF.1609.241.009 of the research programme AiNED XS Europa, which is (partly) financed by the Dutch Research Council (NWO). Additional support was provided by the Cancer Research UK Cambridge Centre [CTRQQR-2021\100012] and The Mark Foundation for Cancer Research [RG95043].

5.0.2 Disclosure of Interests.

The authors have no competing interests to declare that are relevant to the content of this article.