Honest and Reliable Evaluation and Expert Equivalence Testing of Automated Neonatal Seizure Detection

TL;DR

This paper critically evaluates performance metrics for neonatal seizure detection AI, proposing best practices and a multi-rater Turing test to ensure reliable, honest assessment aligned with clinical validation needs.

Contribution

It systematically assesses existing metrics, highlights their limitations, and introduces a comprehensive framework including a multi-rater Turing test for honest AI evaluation in neonatal seizure detection.

Findings

Matthews and Pearson's correlation outperform AUC under class imbalance

Consensus strategies are sensitive to number of raters and agreement levels

Multi-rater Turing test with Fleiss k best captures expert-level AI performance

Abstract

Reliable evaluation of machine learning models for neonatal seizure detection is critical for clinical adoption. Current practices often rely on inconsistent and biased metrics, hindering model comparability and interpretability. Expert-level claims about AI performance are frequently made without rigorous validation, raising concerns about their reliability. This study aims to systematically evaluate common performance metrics and propose best practices tailored to the specific challenges of neonatal seizure detection. Using real and synthetic seizure annotations, we assessed standard performance metrics, consensus strategies, and human-expert level equivalence tests under varying class imbalance, inter-rater agreement, and number of raters. Matthews and Pearson's correlation coefficients outperformed the area under the curve in reflecting performance under class imbalance. Consensus types are sensitive to the number of raters and agreement level among them. Among human-expert level equivalence tests, the multi-rater Turing test using Fleiss k best captured expert-level AI performance. We recommend reporting: (1) at least one balanced metric, (2) Sensitivity, specificity, PPV and NPV, (3) Multi-rater Turing test results using Fleiss k, and (4) All the above on held-out validation set. This proposed framework provides an important prerequisite to clinical validation by enabling a thorough and honest appraisal of AI methods for neonatal seizure detection.