Maximum Likelihood Estimation of Optimal Receiver Operating Characteristic Curves from Likelihood Ratio Observations

TL;DR

This paper introduces a maximum likelihood method to estimate the optimal ROC curve from likelihood ratio samples, providing convergence guarantees and superior accuracy over traditional estimators, especially with limited data.

Contribution

It derives a maximum likelihood estimator for the optimal ROC curve from likelihood ratio observations and proves its convergence and improved accuracy over existing methods.

Findings

MLE converges almost surely to the true ROC curve

MLE outperforms empirical estimators with small sample sizes

The area under the MLE ROC is a consistent estimator of the true area

Abstract

The optimal receiver operating characteristic (ROC) curve, giving the maximum probability of detection as a function of the probability of false alarm, is a key information-theoretic indicator of the difficulty of a binary hypothesis testing problem (BHT). It is well known that the optimal ROC curve for a given BHT, corresponding to the likelihood ratio test, is determined by the probability distribution of the observed data under each of the two hypotheses. In some cases, these two distributions may be unknown or computationally intractable, but independent samples of the likelihood ratio can be observed. This raises the problem of estimating the optimal ROC for a BHT from such samples. The maximum likelihood estimator of the optimal ROC curve is derived, and it is shown to converge almost surely to the true optimal ROC curve in the \levy\ metric, as the number of observations tends to infinity. Finite sample size bounds are obtained for three other estimators: the classical empirical estimator, based on estimating the two types of error probabilities from two separate sets of samples, and two variations of the maximum likelihood estimator called the split estimator and fused estimator, respectively. The maximum likelihood estimator is observed in simulation experiments to be considerably more accurate than the empirical estimator, especially when the number of samples obtained under one of the two hypotheses is small. The area under the maximum likelihood estimator is derived; it is a consistent estimator of the area under the true optimal ROC curve.