Bounding Information Leakage in Machine Learning

TL;DR

This paper introduces a formal framework for understanding information leakage in machine learning models, linking attack success to model generalization and mutual information, with experimental validation on synthetic and real data.

Contribution

It formalizes membership and attribute inference attacks, deriving bounds on attack success and information leakage, and connects these to model generalization and memorization.

Findings

Derived a universal bound on inference attack success rates.

Connected information leakage to the model's generalization gap.

Demonstrated the approach on synthetic and natural image data.

Abstract

Recently, it has been shown that Machine Learning models can leak sensitive information about their training data. This information leakage is exposed through membership and attribute inference attacks. Although many attack strategies have been proposed, little effort has been made to formalize these problems. We present a novel formalism, generalizing membership and attribute inference attack setups previously studied in the literature and connecting them to memorization and generalization. First, we derive a universal bound on the success rate of inference attacks and connect it to the generalization gap of the target model. Second, we study the question of how much sensitive information is stored by the algorithm about its training set and we derive bounds on the mutual information between the sensitive attributes and model parameters. Experimentally, we illustrate the potential of our approach by applying it to both synthetic data and classification tasks on natural images. Finally, we apply our formalism to different attribute inference strategies, with which an adversary is able to recover the identity of writers in the PenDigits dataset.