Real world deployment of a pancreatic cancer risk model: impact of refitting, imputation, and computational burden

TL;DR

This paper evaluates how to effectively deploy a pancreatic cancer risk model in real-world clinical settings, focusing on model refitting and data imputation strategies.

Contribution

The study provides practical guidance on deploying predictive models in clinical settings by comparing model refitting and imputation methods.

Findings

Refitting the model improved discrimination and calibration compared to using the original model.

IFCE imputation offered the best balance between performance and computational efficiency.

Model performance varied across racial and ethnic groups, with poorest calibration among Black patients.

Abstract

Early detection is a major clinical challenge in pancreatic cancer due to its nonspecific symptoms and frequent late-stage diagnosis. While predictive models using electronic health record (EHR) data show promise, their real world implementation remains underexplored. We previously developed a random survival forest (RSF) model to estimate pancreatic cancer risk using structured EHR data from 2007 to 2017. This study evaluates practical considerations for deploying such a model in a prospective clinical context.

We refit the original RSF model using a cohort from 2018 to 2019 and evaluated its performance on a 2020 cohort. We assessed how model refitting and different imputation strategies influenced predictive performance and compared execution times to evaluate computational feasibility. Three imputation strategies were tested: sub-model estimation (SME), stacked multiple imputation (SMI), and imputation via fixed chained equations (IFCE). To simulate real time use, we applied the model to 53 sequential weekly patient batches (with average batch size 190,206).

Refitting improved discrimination and calibration. Without refitting, the C-index ranged from 0.69 to 0.84 depending on imputation method; with refitting, it ranged from 0.79 to 0.83. The IFCE method achieved the best balance between performance (C-index: 0.83 with refit) and runtime (19.54 min). SME had the highest C-index (0.85) and sensitivity (18.41%) but required construction of multiple sub-models. SMI was the most computationally intensive, limiting its scalability in routine use. Calibration improved markedly with refitting. Model performance differed across racial and ethnic groups; calibration was poorest among Black patients but improved with SMI. Execution time varied substantially across methods.

Model refitting and appropriate handling of missing data improve the real world performance of predictive models. Among imputation approaches, IFCE offers the best trade-off between computational efficiency and predictive accuracy. These findings provide practical, implementation-focused guidance for deploying risk prediction models in prospective clinical settings.

Research reported in this publication was supported by the 10.13039/100000054National Cancer Institute of the 10.13039/100000002National Institutes of Health under Award Number R01CA230442.

Figures