Stochastic Negative Mining for Learning with Large Output Spaces

TL;DR

This paper introduces Stochastic Negative Mining, a practical and theoretically grounded method for large output space label retrieval, improving over traditional negative sampling techniques.

Contribution

The paper develops a family of calibrated convex surrogate losses and proposes Stochastic Negative Mining for efficient large-scale label retrieval.

Findings

Stochastic Negative Mining outperforms standard negative sampling methods.

The surrogate losses are shown to be calibrated and convex under certain conditions.

Generalization error bounds are established for the proposed losses.

Abstract

We consider the problem of retrieving the most relevant labels for a given input when the size of the output space is very large. Retrieval methods are modeled as set-valued classifiers which output a small set of classes for each input, and a mistake is made if the label is not in the output set. Despite its practical importance, a statistically principled, yet practical solution to this problem is largely missing. To this end, we first define a family of surrogate losses and show that they are calibrated and convex under certain conditions on the loss parameters and data distribution, thereby establishing a statistical and analytical basis for using these losses. Furthermore, we identify a particularly intuitive class of loss functions in the aforementioned family and show that they are amenable to practical implementation in the large output space setting (i.e. computation is possible without evaluating scores of all labels) by developing a technique called Stochastic Negative Mining. We also provide generalization error bounds for the losses in the family. Finally, we conduct experiments which demonstrate that Stochastic Negative Mining yields benefits over commonly used negative sampling approaches.