Threshold-free Evaluation of Medical Tests for Classification and Prediction: Average Precision versus Area Under the ROC Curve

TL;DR

This paper compares the average precision and AUC metrics for evaluating medical tests, highlighting when AP may be more suitable, and provides practical tools and examples for their application.

Contribution

It offers a mathematical analysis of the relationship between AP and AUC, and derives an asymptotic variance expression for AP in medical test evaluation.

Findings

AP may be more appropriate when early ROC curve performance is important

Derived an expression for the asymptotic variance of AP

Provided real-world examples in cancer biomarker and screening evaluations

Abstract

When evaluating medical tests or biomarkers for disease classification, the area under the receiver-operating characteristic (ROC) curve is a widely used performance metric that does not require us to commit to a specific decision threshold. For the same type of evaluations, a different metric known as the average precision (AP) is used much more widely in the information retrieval literature. We study both metrics in some depths in order to elucidate their difference and relationship. More specifically, we explain mathematically why the AP may be more appropriate if the earlier part of the ROC curve is of interest. We also address practical matters, deriving an expression for the asymptotic variance of the AP, as well as providing real-world examples concerning the evaluation of protein biomarkers for prostate cancer and the assessment of digital versus film mammography for breast cancer screening.