Should I use Synthetic Data for That? An Analysis of the Suitability of Synthetic Data for Data Sharing and Augmentation

TL;DR

This paper critically examines the effectiveness of synthetic data in data sharing and augmentation, identifying fundamental limits and providing guidelines for its appropriate use in various scenarios.

Contribution

It offers a formal analysis and classification of synthetic data use cases, highlighting when synthetic data is suitable or limited for specific data problems.

Findings

Synthetic data often cannot fully replace real data due to inherent limitations.

Many proposed use cases of synthetic data are not well-suited given the identified constraints.

The paper provides decision guidelines for selecting synthetic data solutions.

Abstract

Recent advances in generative modelling have led many to see synthetic data as the go-to solution for a range of problems around data access, scarcity, and under-representation. In this paper, we study three prominent use cases: (1) Sharing synthetic data as a proxy for proprietary datasets to enable statistical analyses while protecting privacy, (2) Augmenting machine learning training sets with synthetic data to improve model performance, and (3) Augmenting datasets with synthetic data to reduce variance in statistical estimation. For each use case, we formalise the problem setting and study, through formal analysis and case studies, under which conditions synthetic data can achieve its intended objectives. We identify fundamental and practical limits that constrain when synthetic data can serve as an effective solution for a particular problem. Our analysis reveals that due to these limits many existing or envisioned use cases of synthetic data are a poor problem fit. Our formalisations and classification of synthetic data use cases enable decision makers to assess whether synthetic data is a suitable approach for their specific data availability problem.