Nearly Optimal Sample Complexity for Learning with Label Proportions

TL;DR

This paper studies Learning from Label Proportions, establishing nearly optimal sample complexity bounds and proposing algorithms that outperform existing methods both theoretically and empirically.

Contribution

It provides the first nearly optimal sample complexity analysis for LLP under square loss and introduces improved algorithms with better empirical performance.

Findings

Sample complexity is essentially optimal and improves on previous bounds.

Algorithms achieve better accuracy with fewer samples in experiments.

Theoretical results show improved dependence on bag size.

Abstract

We investigate Learning from Label Proportions (LLP), a partial information setting where examples in a training set are grouped into bags, and only aggregate label values in each bag are available. Despite the partial observability, the goal is still to achieve small regret at the level of individual examples. We give results on the sample complexity of LLP under square loss, showing that our sample complexity is essentially optimal. From an algorithmic viewpoint, we rely on carefully designed variants of Empirical Risk Minimization, and Stochastic Gradient Descent algorithms, combined with ad hoc variance reduction techniques. On one hand, our theoretical results improve in important ways on the existing literature on LLP, specifically in the way the sample complexity depends on the bag size. On the other hand, we validate our algorithmic solutions on several datasets, demonstrating improved empirical performance (better accuracy for less samples) against recent baselines.