MAPLE: Metadata Augmented Private Language Evolution

TL;DR

MAPLE enhances private language evolution by incorporating domain-specific metadata and in-context learning, leading to better privacy-utility balance, faster convergence, and lower API costs in specialized text generation tasks.

Contribution

It introduces a novel method that uses differentially private metadata and in-context learning to improve the initialization of private language evolution, especially in specialized domains.

Findings

Improves privacy-utility trade-off in domain-specific tasks

Reduces convergence time and API costs

Achieves better performance compared to previous PE methods

Abstract

While differentially private (DP) fine-tuning of large language models (LLMs) is a powerful tool, it is often computationally prohibitive or infeasible when state-of-the-art models are only accessible via proprietary APIs. In such settings, generating DP synthetic data has emerged as a crucial alternative, offering the added benefits of arbitrary reuse across downstream tasks and transparent exploratory data analysis without the opaque constraints of a model's parameter space. Private Evolution (PE) is a promising API-based framework for this goal; however, its performance critically depends on initialization. When the private data distribution deviates substantially from the foundation model's pre-training priors--particularly in highly specialized domains--PE frequently struggles to align with the target data, resulting in degraded utility, poor convergence, and inefficient API usage. To address this initialization bottleneck, we propose Metadata Augmented Private Language Evolution (MAPLE). MAPLE leverages differentially private tabular metadata extraction and in-context learning to effectively ground the initial synthetic distribution in the target domain. Extensive experiments on challenging, domain-specific text generation tasks demonstrate that MAPLE achieves a significantly more favorable privacy-utility trade-off, converges faster, and drastically reduces API costs compared to previous PE methods.