Differentially Private Tabular Data Synthesis using Large Language Models

TL;DR

This paper presents DP-LLMTGen, a novel framework that uses pretrained large language models to generate differentially private synthetic tabular data, improving realism and privacy guarantees.

Contribution

It introduces a two-stage fine-tuning process with a new loss function for LLMs to synthesize private tabular data, outperforming existing methods.

Findings

DP-LLMTGen outperforms existing mechanisms across datasets

The framework demonstrates effective privacy-utility trade-offs

Controllable, fairness-aware data generation is possible

Abstract

Synthetic tabular data generation with differential privacy is a crucial problem to enable data sharing with formal privacy. Despite a rich history of methodological research and development, developing differentially private tabular data generators that can provide realistic synthetic datasets remains challenging. This paper introduces DP-LLMTGen -- a novel framework for differentially private tabular data synthesis that leverages pretrained large language models (LLMs). DP-LLMTGen models sensitive datasets using a two-stage fine-tuning procedure with a novel loss function specifically designed for tabular data. Subsequently, it generates synthetic data through sampling the fine-tuned LLMs. Our empirical evaluation demonstrates that DP-LLMTGen outperforms a variety of existing mechanisms across multiple datasets and privacy settings. Additionally, we conduct an ablation study and several experimental analyses to deepen our understanding of LLMs in addressing this important problem. Finally, we highlight the controllable generation ability of DP-LLMTGen through a fairness-constrained generation setting.