Population and Empirical PR Curves for Assessment of Ranking Algorithms

TL;DR

This paper explores the theoretical and empirical properties of precision-recall (PR) curves, highlighting their differences from ROC curves, especially under various distributional assumptions and data imbalances, to guide proper usage.

Contribution

It introduces a comprehensive analysis of population and empirical PR curves, emphasizing their properties and limitations under different distributional and data conditions.

Findings

Empirical PR curves are inconsistent with discrete score distributions.

Normal approximation fits well for continuous scores but depends on data imbalance.

Distributional assumptions significantly affect PR curve convergence and interpretation.

Abstract

The ROC curve is widely used to assess the quality of prediction/classification/ranking algorithms, and its properties have been extensively studied. The precision-recall (PR) curve has become the de facto replacement for the ROC curve in the presence of imbalance, namely where one class is far more likely than the other class. While the PR and ROC curves tend to be used interchangeably, they have some very different properties. Properties of the PR curve are the focus of this paper. We consider: (1) population PR curves, where complete distributional assumptions are specified for scores from both classes; and (2) empirical estimators of the PR curve, where we observe scores and no distributional assumptions are made. The properties have direct consequence on how the PR curve should, and should not, be used. For example, the empirical PR curve is not consistent when scores in the class of primary interest come from discrete distributions. On the other hand, a normal approximation can fit quite well for points on the empirical PR curve from continuously-defined scores, but convergence can be heavily influenced by the distributional setting, the amount of imbalance, and the point of interest on the PR curve.