Tournament Leave-pair-out Cross-validation for Receiver Operating Characteristic (ROC) Analysis

TL;DR

This paper introduces the tournament leave-pair-out (TLPO) cross-validation method, which accurately estimates AUC and enables ROC analysis, improving bias correction over traditional methods.

Contribution

The paper proposes TLPO, a novel extension of LPO that provides data ranking for ROC analysis while maintaining unbiased AUC estimation.

Findings

TLPO is as reliable as LPO for AUC estimation.

TLPO corrects bias present in leave-one-out cross-validation.

TLPO enables ROC curve analysis with reliable sensitivity and specificity estimates.

Abstract

Receiver operating characteristic (ROC) analysis is widely used for evaluating diagnostic systems. Recent studies have shown that estimating an area under ROC curve (AUC) with standard cross-validation methods suffers from a large bias. The leave-pair-out (LPO) cross-validation has been shown to correct this bias. However, while LPO produces an almost unbiased estimate of AUC, it does not provide a ranking of the data needed for plotting and analyzing the ROC curve. In this study, we propose a new method called tournament leave-pair-out (TLPO) cross-validation. This method extends LPO by creating a tournament from pair comparisons to produce a ranking for the data. TLPO preserves the advantage of LPO for estimating AUC, while it also allows performing ROC analyses. We have shown using both synthetic and real world data that TLPO is as reliable as LPO for AUC estimation, and confirmed the bias in leave-one-out cross-validation on low-dimensional data. As a case study on ROC analysis, we also evaluate how reliably sensitivity and specificity can be estimated from TLPO ROC curves.