Distributed non-disclosive validation of predictive models by a modified ROC-GLM

TL;DR

This paper introduces a privacy-preserving distributed method for validating predictive models using ROC-GLM and differential privacy, enabling AUC calculation without accessing individual data.

Contribution

It develops a novel distributed algorithm for ROC curve and AUC estimation that maintains data privacy using DataSHIELD and differential privacy techniques.

Findings

Algorithm performs well in simulation studies

Successfully applied to multiple sclerosis treatment prediction validation

Maintains data privacy while providing accurate model validation

Abstract

Distributed statistical analyses provide a promising approach for privacy protection when analysing data distributed over several databases. It brings the analysis to the data and not the data to the analysis. The analyst receives anonymous summary statistics which are combined to a aggregated result. We are interested to calculate the AUC of a prediction score based on a distributed approach without getting to know the data of involved individual subjects distributed over different databases. We use DataSHIELD as the technology to carry out distributed analyses and use a newly developed algorithms to perform the validation of the prediction score. Calibration can easily be implemented in the distributed setting. But, discrimination represented by a respective ROC curve and its AUC is challenging. We base our approach on the ROC-GLM algorithm as well as on ideas of differential privacy. The proposed algorithms are evaluated in a simulation study. A real-word application is described: The audit use case of DIFUTURE (Medical Informatics Initiative) with the goal to validate a treatment prediction rule of patients with newly diagnosed multiple sclerosis.