The DCR Delusion: Measuring the Privacy Risk of Synthetic Data

TL;DR

This paper critically evaluates the effectiveness of distance-based metrics like DCR for assessing privacy risks in synthetic data, demonstrating their failures and advocating for membership inference attacks as a more reliable standard.

Contribution

The study reveals that DCR and similar proxy metrics are unreliable for privacy assessment, and emphasizes the need to adopt MIAs for accurate privacy evaluation of synthetic datasets.

Findings

Distance-based metrics often fail to detect privacy leakage.

Datasets deemed private by proxy metrics are vulnerable to MIAs.

Proxy metrics are flawed and miss actual privacy risks.

Abstract

Synthetic data has become an increasingly popular way to share data without revealing sensitive information. Though Membership Inference Attacks (MIAs) are widely considered the gold standard for empirically assessing the privacy of a synthetic dataset, practitioners and researchers often rely on simpler proxy metrics such as Distance to Closest Record (DCR). These metrics estimate privacy by measuring the similarity between the training data and generated synthetic data. This similarity is also compared against that between the training data and a disjoint holdout set of real records to construct a binary privacy test. If the synthetic data is not more similar to the training data than the holdout set is, it passes the test and is considered private. In this work we show that, while computationally inexpensive, DCR and other distance-based metrics fail to identify privacy leakage. Across multiple datasets and both classical models such as Baynet and CTGAN and more recent diffusion models, we show that datasets deemed private by proxy metrics are highly vulnerable to MIAs. We similarly find both the binary privacy test and the continuous measure based on these metrics to be uninformative of actual membership inference risk. We further show that these failures are consistent across different metric hyperparameter settings and record selection methods. Finally, we argue DCR and other distance-based metrics to be flawed by design and show a example of a simple leakage they miss in practice. With this work, we hope to motivate practitioners to move away from proxy metrics to MIAs as the rigorous, comprehensive standard of evaluating privacy of synthetic data, in particular to make claims of datasets being legally anonymous.