Planning for gold: Hypothesis screening with split samples for valid powerful testing in matched observational studies

TL;DR

This paper introduces a novel hypothesis screening method for observational studies that enhances causal inference by splitting data into planning and analysis samples, effectively addressing unmeasured confounding.

Contribution

It proposes a flexible, powerful approach for hypothesis selection in observational studies that accounts for hidden biases and unknown outcome effects.

Findings

Method shows significant benefits in simulations under high unmeasured confounding.

Theoretical analysis confirms the method's robustness and validity.

Applied to flood impact data, it identifies resilient hypotheses for causal inference.

Abstract

Observational studies are valuable tools for inferring causal effects in the absence of controlled experiments. However, these studies may be biased due to the presence of some relevant, unmeasured set of covariates. One approach to mitigate this concern is to identify hypotheses likely to be more resilient to hidden biases by splitting the data into a planning sample for designing the study and an analysis sample for making inferences. We devise a powerful and flexible method for selecting hypotheses in the planning sample when an unknown number of outcomes are affected by the treatment, allowing researchers to gain the benefits of exploratory analysis and still conduct powerful inference under concerns of unmeasured confounding. We investigate the theoretical properties of our method and conduct extensive simulations that demonstrate pronounced benefits, especially at higher levels of allowance for unmeasured confounding. Finally, we demonstrate our method in an observational study of the multi-dimensional impacts of a devastating flood in Bangladesh.