Estimating Model Performance on External Samples from Their Limited Statistical Characteristics

TL;DR

This paper introduces a method to estimate the performance of predictive models on external datasets using limited statistical information, addressing data privacy constraints in healthcare.

Contribution

The paper presents a novel approach that estimates external model performance from limited statistical characteristics, without requiring full dataset access.

Findings

Estimated performance closely matches actual external performance in most cases

Method outperforms internal performance estimates in external validation scenarios

Applicable to healthcare risk models with limited data sharing constraints

Abstract

Methods that address data shifts usually assume full access to multiple datasets. In the healthcare domain, however, privacy-preserving regulations as well as commercial interests limit data availability and, as a result, researchers can typically study only a small number of datasets. In contrast, limited statistical characteristics of specific patient samples are much easier to share and may be available from previously published literature or focused collaborative efforts. Here, we propose a method that estimates model performance in external samples from their limited statistical characteristics. We search for weights that induce internal statistics that are similar to the external ones; and that are closest to uniform. We then use model performance on the weighted internal sample as an estimation for the external counterpart. We evaluate the proposed algorithm on simulated data as well as electronic medical record data for two risk models, predicting complications in ulcerative colitis patients and stroke in women diagnosed with atrial fibrillation. In the vast majority of cases, the estimated external performance is much closer to the actual one than the internal performance. Our proposed method may be an important building block in training robust models and detecting potential model failures in external environments.