Aggregating Data for Optimal and Private Learning

TL;DR

This paper investigates how to optimally partition data into bags for MIR and LLP frameworks to maximize utility in downstream tasks, providing theoretical guarantees and privacy considerations.

Contribution

It introduces a method to find near-optimal data partitioning strategies based on clustering, with utility guarantees and privacy enhancements, applicable to GLMs.

Findings

Optimal bagging reduces to clustering feature vectors or labels.

Theoretical utility guarantees for different loss functions.

Bagging mechanisms can be made label-differentially private.

Abstract

Multiple Instance Regression (MIR) and Learning from Label Proportions (LLP) are learning frameworks arising in many applications, where the training data is partitioned into disjoint sets or bags, and only an aggregate label i.e., bag-label for each bag is available to the learner. In the case of MIR, the bag-label is the label of an undisclosed instance from the bag, while in LLP, the bag-label is the mean of the bag's labels. In this paper, we study for various loss functions in MIR and LLP, what is the optimal way to partition the dataset into bags such that the utility for downstream tasks like linear regression is maximized. We theoretically provide utility guarantees, and show that in each case, the optimal bagging strategy (approximately) reduces to finding an optimal clustering of the feature vectors or the labels with respect to natural objectives such as $k$-means. We also show that our bagging mechanisms can be made label-differentially private, incurring an additional utility error. We then generalize our results to the setting of Generalized Linear Models (GLMs). Finally, we experimentally validate our theoretical results.