Evaluating LLM Simulators as Differentially Private Data Generators

TL;DR

This paper evaluates the ability of LLM-based simulators to generate synthetic data that preserves statistical distributions from differentially private inputs, highlighting their potential and current limitations.

Contribution

It introduces an evaluation of LLM simulators for DP data generation, revealing systematic biases and distribution drift issues that need addressing.

Findings

PersonaLedger achieves AUC 0.70 at epsilon=1 for fraud detection.

LLM biases cause distribution drift, overriding input statistics.

Systematic biases limit LLMs' effectiveness for complex user data.

Abstract

LLM-based simulators offer a promising path for generating complex synthetic data where traditional differentially private (DP) methods struggle with high-dimensional user profiles. But can LLMs faithfully reproduce statistical distributions from DP-protected inputs? We evaluate this using PersonaLedger, an agentic financial simulator, seeded with DP synthetic personas derived from real user statistics. We find that PersonaLedger achieves promising fraud detection utility (AUC 0.70 at epsilon=1) but exhibits significant distribution drift due to systematic LLM biases--learned priors overriding input statistics for temporal and demographic features. These failure modes must be addressed before LLM-based methods can handle the richer user representations where they might otherwise excel.