Properties of the Nonparametric Maximum Likelihood {ROC} Model with a Monotonic Likelihood Ratio

TL;DR

This paper analyzes the properties of a convexity-constrained nonparametric maximum likelihood estimator for ROC curves, showing it is a convex hull of empirical data, with implications for bias and variance in AUC estimation.

Contribution

It provides a direct proof that the convexity-constrained ROC estimator is a convex hull of the empirical ROC curve and evaluates its statistical properties under various conditions.

Findings

The constrained estimator is a convex hull of the empirical ROC curve.

It produces a higher biased AUC estimate but with lower variance.

Standard unbiased variance estimators effectively estimate the variance of the AUC.

Abstract

We expect that some observers in perceptual signal detection experiments, such as radiologists, will make rational decisions, and therefore ratings from those observers are expected to form a convex ROC curve. However, measured and published curves are often not convex. This article examines the convexity-constrained nonparametric maximum likelihood estimator of the ROC curve given by Lloyd (2002). Like Lloyd we use the Pool Adjacent Violator Algorithm (PAVA) to construct the estimate of the convex curve. We present a direct proof that this estimate is a convex hull of the empirical ROC curve. The estimate is simple to construct by hand, and follows the suggestions by Pesce, et~al.~(2010). We examine the properties of this constrained nonparametric maximum likelihood estimator (NPMLE) under a large number of experimental conditions. In particular we examine the behavior of the area under the curve which is often used as summary metric of diagnostic performance. This constrained ROC estimator gives an area under the curve (AUC) estimate that is biased high with respect to the usual empirical AUC estimate, but may be less biased with respect to the underlying continuous true AUC value. The constrained ROC estimator has lower variance than the usual empirical one. Unlike previous authors who used complex bootstrapping to estimate the variance of the constrained NPMLE we demonstrate that standard unbiased estimators of variance work well to estimate the variance of the NPMLE AUC.