Testing weak nulls in matched observational studies

TL;DR

This paper introduces new sensitivity analysis methods for weak null hypotheses in matched observational studies, accommodating treatment effect heterogeneity and improving robustness assessments against hidden biases.

Contribution

It develops valid sensitivity analysis procedures for the weak null in matched designs, including an asymptotically sharp method under certain conditions, advancing robustness testing in observational research.

Findings

The proposed methods are valid for a broad class of test statistics.

Simulations show the alternative procedure maintains validity under various data scenarios.

The methods enable assessment of treatment effect robustness considering effect heterogeneity.

Abstract

We develop sensitivity analyses for weak nulls in matched observational studies while allowing unit-level treatment effects to vary. The methods may be applied to studies using any optimal without-replacement matching algorithm. In contrast to randomized experiments and to paired observational studies, we show for general matched designs that over a large class of test statistics, any valid sensitivity analysis for the entirety of the weak null must be unnecessarily conservative if Fisher's sharp null of no treatment effect for any individual also holds. We present a sensitivity analysis valid for the weak null, and illustrate why it is generally conservative if the sharp null holds through new connections to inverse probability weighted estimators. An alternative procedure is presented that is asymptotically sharp if treatment effects are constant, and that is valid for the weak null under additional restrictions which may be deemed benign by practitioners. Simulations demonstrate that this alternative procedure results in a valid sensitivity analysis for the weak null hypothesis under a host of reasonable data-generating processes. The procedures allow practitioners to assess robustness of estimated sample average treatment effects to hidden bias while allowing for unspecified effect heterogeneity in matched observational studies.