Treatment Effect Bias from Sample Snooping: Blinding Outcomes is Neither Necessary nor Sufficient

TL;DR

This paper demonstrates that outcome blinding in observational studies does not fully prevent bias and introduces an alternative sample partitioning method that allows for unbiased treatment effect estimation without blinding.

Contribution

The paper challenges the common belief that outcome blinding is necessary to prevent bias and proposes a new sample partitioning approach for unbiased effect estimation.

Findings

Outcome blinding is not sufficient to prevent bias when outcomes can be predicted from covariates.

Unblinded analyses can produce bias comparable to blinded ones.

A new sample partitioning method enables unbiased treatment effect estimation without blinding.

Abstract

Popular guidance on observational data analysis states that outcomes should be blinded when determining matching criteria or propensity scores. Such a blinding is informally said to maintain the "objectivity" of the analysis, and to prevent analysts from fishing for positive results by exploiting chance imbalances. Contrary to this notion, we show that outcome blinding is not a sufficient safeguard against fishing. Blinded and unblinded analysts can produce bias of the same order of magnitude in cases where the outcomes can be approximately predicted from baseline covariates. We illustrate this vulnerability with a combination of analytical results and simulations. Finally, to show that outcome blinding is not necessary to prevent bias, we outline an alternative sample partitioning procedure for estimating the average treatment effect on the controls, or the average treatment effect on the treated. This procedure uses all of the the outcome data from all partitions in the final analysis step, but does not require the analysis to not be fully prespecified.