SynQP: A Framework and Metrics for Evaluating the Quality and Privacy Risk of Synthetic Data

TL;DR

SynQP is an open framework designed to benchmark privacy risks in synthetic data generation, especially for health applications, using simulated data and new metrics to improve evaluation transparency and safety.

Contribution

The paper introduces SynQP, a novel open framework and metrics for privacy evaluation of synthetic data, addressing the lack of benchmark datasets and fair privacy assessment methods.

Findings

SynQP enables effective benchmarking of privacy risks in synthetic data.

The proposed identity disclosure risk metric provides more accurate privacy risk estimation.

Differential privacy consistently reduces privacy risks below regulatory thresholds.

Abstract

The use of synthetic data in health applications raises privacy concerns, yet the lack of open frameworks for privacy evaluations has slowed its adoption. A major challenge is the absence of accessible benchmark datasets for evaluating privacy risks, due to difficulties in acquiring sensitive data. To address this, we introduce SynQP, an open framework for benchmarking privacy in synthetic data generation (SDG) using simulated sensitive data, ensuring that original data remains confidential. We also highlight the need for privacy metrics that fairly account for the probabilistic nature of machine learning models. As a demonstration, we use SynQP to benchmark CTGAN and propose a new identity disclosure risk metric that offers a more accurate estimation of privacy risks compared to existing approaches. Our work provides a critical tool for improving the transparency and reliability of privacy evaluations, enabling safer use of synthetic data in health-related applications. % In our quality evaluations, non-private models achieved near-perfect machine-learning efficacy \(\ge0.97\). Our privacy assessments (Table II) reveal that DP consistently lowers both identity disclosure risk (SD-IDR) and membership-inference attack risk (SD-MIA), with all DP-augmented models staying below the 0.09 regulatory threshold. Code available at https://github.com/CAN-SYNH/SynQP