Assessing Utility of Differential Privacy for RCTs

TL;DR

This paper evaluates how differential privacy techniques affect the analysis of randomized controlled trial data, demonstrating that privacy-preserving methods can maintain valid statistical inference in published research.

Contribution

It empirically assesses the impact of differential privacy algorithms on RCT analysis, highlighting necessary adjustments and demonstrating practical applicability.

Findings

Privacy-preserving methods can support valid statistical inference in RCTs.

Adjustments are needed for stability-based histogram algorithms.

Simulation studies confirm the feasibility of privacy-protected analysis.

Abstract

Randomized controlled trials (RCTs) have become powerful tools for assessing the impact of interventions and policies in many contexts. They are considered the gold standard for causal inference in the biomedical fields and many social sciences. Researchers have published an increasing number of studies that rely on RCTs for at least part of their inference. These studies typically include the response data that has been collected, de-identified, and sometimes protected through traditional disclosure limitation methods. In this paper, we empirically assess the impact of privacy-preserving synthetic data generation methodologies on published RCT analyses by leveraging available replication packages (research compendia) in economics and policy analysis. We implement three privacy-preserving algorithms, that use as a base one of the basic differentially private (DP) algorithms, the perturbed histogram, to support the quality of statistical inference. We highlight challenges with the straight use of this algorithm and the stability-based histogram in our setting and described the adjustments needed. We provide simulation studies and demonstrate that we can replicate the analysis in a published economics article on privacy-protected data under various parameterizations. We find that relatively straightforward (at a high-level) privacy-preserving methods influenced by DP techniques allow for inference-valid protection of published data. The results have applicability to researchers wishing to share RCT data, especially in the context of low- and middle-income countries, with strong privacy protection.