Probably Approximately Precision and Recall Learning

TL;DR

This paper introduces a PAC framework for learning precision and recall metrics in settings with positive-only feedback, proposing new algorithms that outperform classical methods in such challenging scenarios.

Contribution

It extends traditional learning models to set-based hypotheses and develops algorithms that learn effectively from positive-only data, with optimal sample complexity and approximation guarantees.

Findings

Classical ERM methods fail under positive-only feedback.

New algorithms achieve optimal sample complexity in realizable cases.

Agnostic case algorithms provide multiplicative approximation guarantees.

Abstract

Precision and Recall are fundamental metrics in machine learning tasks where both accurate predictions and comprehensive coverage are essential, such as in multi-label learning, language generation, medical studies, and recommender systems. A key challenge in these settings is the prevalence of one-sided feedback, where only positive examples are observed during training--e.g., in multi-label tasks like tagging people in Facebook photos, we may observe only a few tagged individuals, without knowing who else appears in the image. To address learning under such partial feedback, we introduce a Probably Approximately Correct (PAC) framework in which hypotheses are set functions that map each input to a set of labels, extending beyond single-label predictions and generalizing classical binary, multi-class, and multi-label models. Our results reveal sharp statistical and algorithmic separations from standard settings: classical methods such as Empirical Risk Minimization provably fail, even for simple hypothesis classes. We develop new algorithms that learn from positive data alone, achieving optimal sample complexity in the realizable case, and establishing multiplicative--rather than additive-approximation guarantees in the agnostic case, where achieving additive regret is impossible.