Generative Models for Effective ML on Private, Decentralized Datasets

TL;DR

This paper introduces differentially private federated generative models that enable effective data debugging and analysis in privacy-sensitive and decentralized settings, where raw data cannot be directly accessed.

Contribution

It presents novel federated generative modeling techniques with formal privacy guarantees for debugging and analyzing private, decentralized datasets.

Findings

Generative models can identify data issues without direct data access.

Federated RNNs and GANs with differential privacy are effective for data debugging.

Methods work on both text and image datasets.

Abstract

To improve real-world applications of machine learning, experienced modelers develop intuition about their datasets, their models, and how the two interact. Manual inspection of raw data - of representative samples, of outliers, of misclassifications - is an essential tool in a) identifying and fixing problems in the data, b) generating new modeling hypotheses, and c) assigning or refining human-provided labels. However, manual data inspection is problematic for privacy sensitive datasets, such as those representing the behavior of real-world individuals. Furthermore, manual data inspection is impossible in the increasingly important setting of federated learning, where raw examples are stored at the edge and the modeler may only access aggregated outputs such as metrics or model parameters. This paper demonstrates that generative models - trained using federated methods and with formal differential privacy guarantees - can be used effectively to debug many commonly occurring data issues even when the data cannot be directly inspected. We explore these methods in applications to text with differentially private federated RNNs and to images using a novel algorithm for differentially private federated GANs.