Dissecting Distribution Inference

TL;DR

This paper introduces a new black-box attack for distribution inference that outperforms existing white-box methods, evaluates various defenses, and finds simple re-sampling defenses to be highly effective in protecting data privacy.

Contribution

The paper develops a novel black-box attack for distribution inference, evaluates its effectiveness under relaxed assumptions, and proposes a simple yet effective re-sampling defense.

Findings

The new attack outperforms previous white-box attacks in most settings.

Noise-based defenses are largely ineffective against the new attack.

Re-sampling defenses can significantly mitigate distribution inference risks.

Abstract

A distribution inference attack aims to infer statistical properties of data used to train machine learning models. These attacks are sometimes surprisingly potent, but the factors that impact distribution inference risk are not well understood and demonstrated attacks often rely on strong and unrealistic assumptions such as full knowledge of training environments even in supposedly black-box threat scenarios. To improve understanding of distribution inference risks, we develop a new black-box attack that even outperforms the best known white-box attack in most settings. Using this new attack, we evaluate distribution inference risk while relaxing a variety of assumptions about the adversary's knowledge under black-box access, like known model architectures and label-only access. Finally, we evaluate the effectiveness of previously proposed defenses and introduce new defenses. We find that although noise-based defenses appear to be ineffective, a simple re-sampling defense can be highly effective. Code is available at https://github.com/iamgroot42/dissecting_distribution_inference