Assessing binary classifiers using only positive and unlabeled data

TL;DR

This paper introduces a method to evaluate binary classifiers using only positive and unlabeled data by estimating performance metrics like ROC and PR curves through the fraction of positives in unlabeled data.

Contribution

It presents a novel approach to estimate classifier performance metrics solely from positive and unlabeled data, with theoretical bounds and empirical validation.

Findings

Reliable estimation of ROC and PR curves from positive and unlabeled data

Theoretical bounds on estimation quality

Empirical validation on real datasets

Abstract

Assessing the performance of a learned model is a crucial part of machine learning. However, in some domains only positive and unlabeled examples are available, which prohibits the use of most standard evaluation metrics. We propose an approach to estimate any metric based on contingency tables, including ROC and PR curves, using only positive and unlabeled data. Estimating these performance metrics is essentially reduced to estimating the fraction of (latent) positives in the unlabeled set, assuming known positives are a random sample of all positives. We provide theoretical bounds on the quality of our estimates, illustrate the importance of estimating the fraction of positives in the unlabeled set and demonstrate empirically that we are able to reliably estimate ROC and PR curves on real data.