Membership Inference Attacks against Synthetic Data through Overfitting Detection

TL;DR

This paper introduces DOMIAS, a density-based membership inference attack that effectively detects whether real data was used in training generative models, highlighting privacy risks especially for underrepresented groups.

Contribution

The work proposes a realistic MIA setting and introduces DOMIAS, a novel density-based attack that outperforms previous methods in detecting membership, particularly for uncommon samples.

Findings

DOMIAS significantly improves MIA success over prior methods.

It is especially effective on underrepresented, uncommon samples.

Provides an interpretable privacy metric for synthetic data.

Abstract

Data is the foundation of most science. Unfortunately, sharing data can be obstructed by the risk of violating data privacy, impeding research in fields like healthcare. Synthetic data is a potential solution. It aims to generate data that has the same distribution as the original data, but that does not disclose information about individuals. Membership Inference Attacks (MIAs) are a common privacy attack, in which the attacker attempts to determine whether a particular real sample was used for training of the model. Previous works that propose MIAs against generative models either display low performance -- giving the false impression that data is highly private -- or need to assume access to internal generative model parameters -- a relatively low-risk scenario, as the data publisher often only releases synthetic data, not the model. In this work we argue for a realistic MIA setting that assumes the attacker has some knowledge of the underlying data distribution. We propose DOMIAS, a density-based MIA model that aims to infer membership by targeting local overfitting of the generative model. Experimentally we show that DOMIAS is significantly more successful at MIA than previous work, especially at attacking uncommon samples. The latter is disconcerting since these samples may correspond to underrepresented groups. We also demonstrate how DOMIAS' MIA performance score provides an interpretable metric for privacy, giving data publishers a new tool for achieving the desired privacy-utility trade-off in their synthetic data.