Priming bias versus post-treatment bias in experimental designs

TL;DR

This paper explores the trade-off between priming bias and post-treatment bias in experimental designs, providing bounds and methods to assess their impact on causal inference.

Contribution

It offers a formal analysis of biases in different measurement timings and introduces bounds to evaluate their effects on causal estimates.

Findings

Derived nonparametric bounds for treatment-moderator interactions

Showed how to narrow bounds using substantive assumptions

Applied methodology to electoral messaging survey

Abstract

Conditioning on variables affected by treatment can induce post-treatment bias when estimating causal effects. Although this suggests that researchers should measure potential moderators before administering the treatment in an experiment, doing so may also bias causal effect estimation if the covariate measurement primes respondents to react differently to the treatment. This paper formally analyzes this trade-off between post-treatment and priming biases in three experimental designs that vary when moderators are measured: pre-treatment, post-treatment, or a randomized choice between the two. We derive nonparametric bounds for interactions between the treatment and the moderator under each design and show how to use substantive assumptions to narrow these bounds. These bounds allow researchers to assess the sensitivity of their empirical findings to priming and post-treatment bias. We then apply the proposed methodology to a survey experiment on electoral messaging.