Quantifying Uncertainty in Classification Performance: ROC Confidence Bands Using Conformal Prediction

TL;DR

This paper introduces a conformal prediction-based algorithm to construct confidence bands for ROC curves, enabling reliable uncertainty quantification in classification performance assessments.

Contribution

It presents a novel method for creating ROC confidence bands that account for data randomness, applicable to both iid and non-iid test data scenarios.

Findings

The method provides valid coverage probabilities for ROC confidence bands.

The approach is validated through theoretical proofs and numerical experiments.

It improves the reliability of ROC analysis in uncertain data conditions.

Abstract

To evaluate a classification algorithm, it is common practice to plot the ROC curve using test data. However, the inherent randomness in the test data can undermine our confidence in the conclusions drawn from the ROC curve, necessitating uncertainty quantification. In this article, we propose an algorithm to construct confidence bands for the ROC curve, quantifying the uncertainty of classification on the test data in terms of sensitivity and specificity. The algorithm is based on a procedure called conformal prediction, which constructs individualized confidence intervals for the test set and the confidence bands for the ROC curve can be obtained by combining the individualized intervals together. Furthermore, we address both scenarios where the test data are either iid or non-iid relative to the observed data set and propose distinct algorithms for each case with valid coverage probability. The proposed method is validated through both theoretical results and numerical experiments.