SoK: Can Synthetic Images Replace Real Data? A Survey of Utility and Privacy of Synthetic Image Generation

TL;DR

This survey evaluates whether synthetic images can replace real data by systematically analyzing various generation methods, privacy risks, and utility-privacy tradeoffs through empirical benchmarking and attack assessments.

Contribution

It provides a comprehensive categorization of synthetic image generation techniques, benchmarks multiple methods, and assesses privacy risks using membership inference attacks.

Findings

Synthetic images can sometimes effectively replace real data depending on the method.

Privacy mitigations can improve privacy but may reduce data utility.

Certain generative models outperform others in balancing utility and privacy.

Abstract

Advances in generative models have transformed the field of synthetic image generation for privacy-preserving data synthesis (PPDS). However, the field lacks a comprehensive survey and comparison of synthetic image generation methods across diverse settings. In particular, when we generate synthetic images for the purpose of training a classifier, there is a pipeline of generation-sampling-classification which takes private training as input and outputs the final classifier of interest. In this survey, we systematically categorize existing image synthesis methods, privacy attacks, and mitigations along this generation-sampling-classification pipeline. To empirically compare diverse synthesis approaches, we provide a benchmark with representative generative methods and use model-agnostic membership inference attacks (MIAs) as a measure of privacy risk. Through this study, we seek to answer critical questions in PPDS: Can synthetic data effectively replace real data? Which release strategy balances utility and privacy? Do mitigations improve the utility-privacy tradeoff? Which generative models perform best across different scenarios? With a systematic evaluation of diverse methods, our study provides actionable insights into the utility-privacy tradeoffs of synthetic data generation methods and guides the decision on optimal data releasing strategies for real-world applications.