Balancing Covariates in Survey Experiments

TL;DR

This paper proposes a stratified and rerandomized sampling method for survey experiments to improve covariate balance and estimation efficiency, supported by asymptotic theory and numerical validation.

Contribution

It introduces a novel stratified rerandomization design and develops a theoretical framework for its asymptotic properties, enhancing covariate balance in survey experiments.

Findings

The proposed design achieves better covariate balance than standard methods.

The asymptotic distribution of the estimator is more concentrated at the true effect.

Numerical studies confirm the efficiency gains of the new method.

Abstract

The survey experiment is widely used in economics and social sciences to evaluate the effects of treatments or programs. In a standard population-based survey experiment, the experimenter randomly draws experimental units from a target population of interest and then randomly assigns the sampled units to treatment or control conditions to explore the treatment effect of an intervention. Simple random sampling and treatment assignment can balance covariates on average. However, covariate imbalance often exists in finite samples. To address the imbalance issue, we study a stratified approach to balance covariates in a survey experiment. A stratified rejective sampling and rerandomization design is further proposed to enhance the covariate balance. We develop a design-based asymptotic theory for the widely used stratified difference-in-means estimator of the average treatment effect under the proposed design. In particular, we show that it is consistent and asymptotically a convolution of a normal distribution and two truncated normal distributions. This limiting distribution is more concentrated at the true average treatment effect than that under the existing experimental designs. Moreover, we propose a covariate adjustment method in the analysis stage, which can further improve the estimation efficiency. Numerical studies demonstrate the validity and improved efficiency of the proposed method.