When Privacy Isn't Synthetic: Hidden Data Leakage in Generative AI Models

TL;DR

This paper reveals that generative AI models can leak sensitive training data through structural overlaps in synthetic outputs, even when privacy-preserving techniques are used, highlighting a new privacy risk.

Contribution

It introduces a black-box membership inference attack exploiting data distribution overlaps, demonstrating privacy vulnerabilities in synthetic data generation.

Findings

Membership leakage occurs despite differential privacy.

Clustering reveals high-density regions linked to training data.

Attack is effective across multiple sensitive domains.

Abstract

Generative models are increasingly used to produce privacy-preserving synthetic data as a safe alternative to sharing sensitive training datasets. However, we demonstrate that such synthetic releases can still leak information about the underlying training samples through structural overlap in the data manifold. We propose a black-box membership inference attack that exploits this vulnerability without requiring access to model internals or real data. The attacker repeatedly queries the generative model to obtain large numbers of synthetic samples, performs unsupervised clustering to identify dense regions of the synthetic distribution, and then analyzes cluster medoids and neighborhoods that correspond to high-density regions in the original training data. These neighborhoods act as proxies for training samples, enabling the adversary to infer membership or reconstruct approximate records. Our experiments across healthcare, finance, and other sensitive domains show that cluster overlap between real and synthetic data leads to measurable membership leakage-even when the generator is trained with differential privacy or other noise mechanisms. The results highlight an under-explored attack surface in synthetic data generation pipelines and call for stronger privacy guarantees that account for distributional neighborhood inference rather than sample-level memorization alone, underscoring its role in privacy-preserving data publishing. Implementation and evaluation code are publicly available at:github.com/Cluster-Medoid-Leakage-Attack.