Empirical investigation of multi-source cross-validation in clinical ECG classification

TL;DR

This study empirically compares cross-validation methods in multi-source ECG classification, revealing that leave-source-out cross-validation offers more realistic performance estimates than standard K-fold methods, which tend to be overly optimistic.

Contribution

It systematically evaluates cross-validation strategies in multi-source medical data, demonstrating the advantages of leave-source-out validation for realistic performance assessment.

Findings

K-fold cross-validation overestimates accuracy for new sources.

Leave-source-out validation provides less biased estimates.

Multi-source data improves evaluation reliability.

Abstract

Traditionally, machine learning-based clinical prediction models have been trained and evaluated on patient data from a single source, such as a hospital. Cross-validation methods can be used to estimate the accuracy of such models on new patients originating from the same source, by repeated random splitting of the data. However, such estimates tend to be highly overoptimistic when compared to accuracy obtained from deploying models to sources not represented in the dataset, such as a new hospital. The increasing availability of multi-source medical datasets provides new opportunities for obtaining more comprehensive and realistic evaluations of expected accuracy through source-level cross-validation designs. In this study, we present a systematic empirical evaluation of standard K-fold cross-validation and leave-source-out cross-validation methods in a multi-source setting. We consider the task of electrocardiogram based cardiovascular disease classification, combining and harmonizing the openly available PhysioNet CinC Challenge 2021 and the Shandong Provincial Hospital datasets for our study. Our results show that K-fold cross-validation, both on single-source and multi-source data, systemically overestimates prediction performance when the end goal is to generalize to new sources. Leave-source-out cross-validation provides more reliable performance estimates, having close to zero bias though larger variability. The evaluation highlights the dangers of obtaining misleading cross-validation results on medical data and demonstrates how these issues can be mitigated when having access to multi-source data.