Rethinking external validation for the target population: Capturing patient-level similarity with a generative model

TL;DR

This paper introduces a generative model-based framework to improve external validation of predictive models by assessing patient similarity and population differences, enhancing interpretability and transportability insights.

Contribution

It proposes a novel autoencoder-based similarity measure that distinguishes model deficiencies from case-mix effects without sharing original data.

Findings

Substantial performance variation across similarity-defined subgroups.

Conventional validation can mask clinically relevant performance deficits.

Framework links model performance to population alignment, aiding transportability decisions.

Abstract

Background: External validation is essential for assessing the transportability of predictive models. However, its interpretation is often confounded by differences between external and development populations. This study introduces a framework to distinguish model deficiencies from case-mix effects. Method: We propose a framework that quantifies each external patient's similarity to the development data and measures performance in subgroups with varying levels of alignment to the development distribution. We use generative models, specifically autoencoders, to estimate similarity, offering a more flexible alternative to traditional linear approaches and enabling validation without sharing the original development data. The utility of autoencoder-based similarity measure is demonstrated using synthetic data, and the framework's application is illustrated using data from the Netherlands Heart Registration (NHR) to predict mortality after transcatheter aortic valve implantation. Results: Our framework revealed substantial variation in model performance across similarity-defined subgroups, differences that remain hidden under conventional external validation yet can meaningfully alter conclusions. In several settings, conventional external validation suggested poor overall performance. However, after accounting for differences in patient characteristics, for some sub-groups, the model performance was consistent with internal validation results. Conversely, apparently acceptable overall performance could mask clinically relevant performance deficits in specific subgroups. Conclusion: The proposed framework enhances the interpretability of external validation by linking model performance to population alignment with the development data. This provides a more principled basis for deciding whether a model is transportable and to which patients it can be safely applied.