Empirical Evaluation of Structured Synthetic Data Privacy Metrics: Novel experimental framework

TL;DR

This paper introduces an experimental framework to empirically evaluate synthetic data privacy metrics, enabling better benchmarking and understanding of privacy protection in synthetic data generation.

Contribution

It proposes a novel framework for empirical assessment of synthetic data privacy metrics and applies it to existing methods using controlled experiments.

Findings

Framework effectively benchmarks privacy quantification methods.

Survey of existing approaches and legal context provided.

Applied framework to main privacy methods with no-box threat models.

Abstract

Synthetic data generation is gaining traction as a privacy enhancing technology (PET). When properly generated, synthetic data preserve the analytic utility of real data while avoiding the retention of information that would allow the identification of specific individuals. However, the concept of data privacy remains elusive, making it challenging for practitioners to evaluate and benchmark the degree of privacy protection offered by synthetic data. In this paper, we propose a framework to empirically assess the efficacy of tabular synthetic data privacy quantification methods through controlled, deliberate risk insertion. To demonstrate this framework, we survey existing approaches to synthetic data privacy quantification and the related legal theory. We then apply the framework to the main privacy quantification methods with no-box threat models on publicly available datasets.