The Sample Complexity of Membership Inference and Privacy Auditing

TL;DR

This paper investigates the minimum number of reference samples needed for successful membership inference attacks, revealing that attackers sometimes require significantly more data than the training set, impacting privacy assessments.

Contribution

It provides the first analysis of sample complexity for membership inference in Gaussian models, showing the potential for more powerful attacks with additional distribution knowledge.

Findings

Attacker may need more samples than training data for successful inference.

Current practical attacks are limited to O(n) samples, possibly underestimating risks.

More information about the data distribution enables stronger membership inference attacks.

Abstract

A membership-inference attack gets the output of a learning algorithm, and a target individual, and tries to determine whether this individual is a member of the training data or an independent sample from the same distribution. A successful membership-inference attack typically requires the attacker to have some knowledge about the distribution that the training data was sampled from, and this knowledge is often captured through a set of independent reference samples from that distribution. In this work we study how much information the attacker needs for membership inference by investigating the sample complexity-the minimum number of reference samples required-for a successful attack. We study this question in the fundamental setting of Gaussian mean estimation where the learning algorithm is given $n$ samples from a Gaussian distribution $\mathcal{N}(\mu,\Sigma)$ in $d$ dimensions, and tries to estimate $\hat\mu$ up to some error $\mathbb{E}[\|\hat \mu - \mu\|^2_{\Sigma}]\leq \rho^2 d$. Our result shows that for membership inference in this setting, $\Omega(n + n^2 \rho^2)$ samples can be necessary to carry out any attack that competes with a fully informed attacker. Our result is the first to show that the attacker sometimes needs many more samples than the training algorithm uses to train the model. This result has significant implications for practice, as all attacks used in practice have a restricted form that uses $O(n)$ samples and cannot benefit from $\omega(n)$ samples. Thus, these attacks may be underestimating the possibility of membership inference, and better attacks may be possible when information about the distribution is easy to obtain.