# Prospective validation and real-time implementation of an automated machine learning postoperative mortality prediction model

## TL;DR

This study validated a machine learning model for predicting postoperative mortality in real-time and showed it can be integrated into electronic health records.

## Contribution

The paper demonstrates the real-time implementation and prospective validation of a machine learning model in a clinical setting.

## Key findings

- The implemented model achieved an AUROC of 0.874 in predicting postoperative mortality.
- Real-time data updates and automated model output transfer to the EHR were successfully demonstrated.
- The model's performance was comparable to the original 58-feature version and better than ASA physical status.

## Abstract

Machine learning prediction models require prospective validation to ensure implementation fidelity and feasibility. Our primary objective was to prospectively validate a previously reported postoperative mortality prediction model in inpatients undergoing surgery. Our secondary objective was to evaluate feasibility of a pilot clinical decision support tool.

We prospectively validated and implemented a random forest machine learning model trained to predict in-hospital mortality using data from a single academic medical centre. A reduced 32-feature model was implemented into the electronic health record (EHR) using a real-time data mart at the same institution. To assess model performance, the area under the receiver operating characteristic curve (AUROC), area under the curve precision-recall (AUCPR), and other performance measures were calculated. To assess feasibility, implementation workflow metrics were evaluated and a survey was administered to anaesthesiologists trained to use the pilot clinical decision support tool.

The AUROC for the prospectively implemented model was 0.874 (95% confidence interval [CI] 0.860—0.887), and the AUCPR was 0.111. By comparison, the AUROC for the 58-feature model was 0.925 (95% CI 0.900—0.947), and for ASA physical status the AUROC was 0.814 (95% CI 0.802—0.827) and the AUCPR was 0.103. The implementation demonstrated feasibility through real-time data updates, automated transfer of model outputs to the EHR, and provider survey entries.

This prospective validation and EHR implementation of a previously published random forest machine learning model predicting postoperative in-hospital mortality demonstrated acceptable real-world performance of the implemented model and feasibility of integrating such a system into clinical practice.