A Technical Exploration of Causal Inference with Hybrid LLM Synthetic Data

TL;DR

This paper explores how to generate synthetic data using Large Language Models that accurately preserves causal effects, proposing a hybrid framework to improve causal inference reliability in synthetic datasets.

Contribution

The paper introduces a hybrid data generation framework combining model-based covariate synthesis with causal structure preservation, enhancing causal effect estimation in synthetic data.

Findings

State-of-the-art synthetic generators often misestimate causal effects.

The proposed hybrid framework improves causal effect preservation.

Benchmarking shows better causal inference performance with the new method.

Abstract

Large Language Models (LLMs) offer a flexible means to generate synthetic tabular data, yet existing approaches often fail to preserve key causal parameters such as the average treatment effect (ATE). In this technical exploration, we first demonstrate that state-of-the-art synthetic data generators, both GAN- and LLM-based, can achieve high predictive fidelity while substantially misestimating causal effects. To address this gap, we propose a hybrid generation framework that combines model-based covariate synthesis (monitored via distance-to-closest-record filtering) with separately learned propensity and outcome models, thereby ensuring that (W, A, Y) triplets retain their underlying causal structure. We further introduce a synthetic pairing strategy to mitigate positivity violations and a realistic evaluation protocol that leverages unlimited synthetic samples to benchmark traditional estimators (IPTW, AIPW, substitution) under complex covariate distributions. This work lays the groundwork for LLM-powered data pipelines that support robust causal analysis. Our code is available at https://github.com/Xyc-arch/llm-synthetic-for-causal-inference.git.