Learning from Aggregated Data: Curated Bags versus Random Bags

TL;DR

This paper investigates training machine learning models using aggregated data labels, comparing curated and random bag grouping methods, and demonstrates that privacy-preserving aggregation can maintain model performance through theoretical analysis and empirical validation.

Contribution

It introduces methods for gradient-based learning with aggregated labels in curated bags without performance loss and provides risk bounds for random bags, advancing privacy-preserving machine learning techniques.

Findings

Curated bags allow gradient-based learning without performance degradation.

Risk bounds for random bags depend on bag size and hypothesis complexity.

Empirical results support the effectiveness of aggregate learning for privacy and accuracy.

Abstract

Protecting user privacy is a major concern for many machine learning systems that are deployed at scale and collect from a diverse set of population. One way to address this concern is by collecting and releasing data labels in an aggregated manner so that the information about a single user is potentially combined with others. In this paper, we explore the possibility of training machine learning models with aggregated data labels, rather than individual labels. Specifically, we consider two natural aggregation procedures suggested by practitioners: curated bags where the data points are grouped based on common features and random bags where the data points are grouped randomly in bag of similar sizes. For the curated bag setting and for a broad range of loss functions, we show that we can perform gradient-based learning without any degradation in performance that may result from aggregating data. Our method is based on the observation that the sum of the gradients of the loss function on individual data examples in a curated bag can be computed from the aggregate label without the need for individual labels. For the random bag setting, we provide a generalization risk bound based on the Rademacher complexity of the hypothesis class and show how empirical risk minimization can be regularized to achieve the smallest risk bound. In fact, in the random bag setting, there is a trade-off between size of the bag and the achievable error rate as our bound indicates. Finally, we conduct a careful empirical study to confirm our theoretical findings. In particular, our results suggest that aggregate learning can be an effective method for preserving user privacy while maintaining model accuracy.