Valid Inference with Imperfect Synthetic Data

TL;DR

This paper introduces a new statistically valid estimator for combining real and synthetic data generated by large language models, with strong theoretical guarantees and empirical validation in social science applications.

Contribution

A novel hyperparameter-free estimator based on generalized method of moments that effectively integrates synthetic and real data for valid inference.

Findings

Interactions between synthetic and real data residuals improve estimates

The estimator achieves strong theoretical guarantees

Empirical results show significant performance gains

Abstract

Predictions and generations from large language models are increasingly being explored as an aid in limited data regimes, such as in computational social science and human subjects research. While prior technical work has mainly explored the potential to use model-predicted labels for unlabeled data in a principled manner, there is increasing interest in using large language models to generate entirely new synthetic samples (e.g., synthetic simulations), such as in responses to surveys. However, it remains unclear by what means practitioners can combine such data with real data and yet produce statistically valid conclusions upon them. In this paper, we introduce a new estimator based on generalized method of moments, providing a hyperparameter-free solution with strong theoretical guarantees to address this challenge. Intriguingly, we find that interactions between the moment residuals of synthetic data and those of real data (i.e., when they are predictive of each other) can greatly improve estimates of the target parameter. We validate the finite-sample performance of our estimator across different tasks in computational social science applications, demonstrating large empirical gains.