On Primes, Log-Loss Scores and (No) Privacy

TL;DR

This paper demonstrates that exposing Log-Loss scores in privacy auditing tools allows adversaries to achieve perfect membership inference, breaching privacy even without model-specific attack training or side knowledge.

Contribution

It proves that Log-Loss scores enable full membership inference accuracy in a single query, regardless of model memorization or overfitting, highlighting privacy risks of statistical aggregates.

Findings

Log-Loss scores enable perfect membership inference.

Adversaries do not need attack training or side knowledge.

Privacy leakage occurs even with non-overfitting models.

Abstract

Membership Inference Attacks exploit the vulnerabilities of exposing models trained on customer data to queries by an adversary. In a recently proposed implementation of an auditing tool for measuring privacy leakage from sensitive datasets, more refined aggregates like the Log-Loss scores are exposed for simulating inference attacks as well as to assess the total privacy leakage based on the adversary's predictions. In this paper, we prove that this additional information enables the adversary to infer the membership of any number of datapoints with full accuracy in a single query, causing complete membership privacy breach. Our approach obviates any attack model training or access to side knowledge with the adversary. Moreover, our algorithms are agnostic to the model under attack and hence, enable perfect membership inference even for models that do not memorize or overfit. In particular, our observations provide insight into the extent of information leakage from statistical aggregates and how they can be exploited.