Regret Bounds for Non-decomposable Metrics with Missing Labels

TL;DR

This paper develops a framework for optimizing non-decomposable metrics like F1 in settings with missing labels, providing theoretical regret bounds and demonstrating improved empirical performance.

Contribution

It introduces a generic approach to handle missing labels in non-decomposable metric optimization with provable regret bounds across multiple learning settings.

Findings

Regret bounds are derived for collaborative filtering, multilabel classification, and PU learning.

The proposed method outperforms existing approaches that ignore missing label information.

Empirical results show significant improvements in F1 score on synthetic and benchmark datasets.

Abstract

We consider the problem of recommending relevant labels (items) for a given data point (user). In particular, we are interested in the practically important setting where the evaluation is with respect to non-decomposable (over labels) performance metrics like the $F_1$ measure, and the training data has missing labels. To this end, we propose a generic framework that given a performance metric $\Psi$, can devise a regularized objective function and a threshold such that all the values in the predicted score vector above and only above the threshold are selected to be positive. We show that the regret or generalization error in the given metric $\Psi$ is bounded ultimately by estimation error of certain underlying parameters. In particular, we derive regret bounds under three popular settings: a) collaborative filtering, b) multilabel classification, and c) PU (positive-unlabeled) learning. For each of the above problems, we can obtain precise non-asymptotic regret bound which is small even when a large fraction of labels is missing. Our empirical results on synthetic and benchmark datasets demonstrate that by explicitly modeling for missing labels and optimizing the desired performance metric, our algorithm indeed achieves significantly better performance (like $F_1$ score) when compared to methods that do not model missing label information carefully.