Detecting Where Effects Occur by Testing Hypotheses in Order

TL;DR

This paper introduces a top-down hierarchical testing procedure to identify where effects occur in multi-site experiments, significantly improving detection power over standard methods while maintaining error control.

Contribution

It develops a tree-based hypothesis testing method that enhances detection of effects in specific groups or sites, with valid error control and adaptive adjustments.

Findings

Detects effects in 44% of non-null blocks versus 11% with standard methods.

Provides a diagnostic for when no adjustment is needed based on power decay.

Applied to 25 education trials with an available R package.

Abstract

Experimental evaluations of public policies often randomize a new intervention within many sites or blocks. After a report of an overall result -- statistically significant or not -- the natural question from a policy maker is: \emph{where} did any effects occur? Standard adjustments for multiple testing provide little power to answer this question. In simulations modeled after a 44-block education trial, the Hommel adjustment -- among the most powerful procedures controlling the family-wise error rate (FWER) -- detects effects in only 11\% of truly non-null blocks. We develop a procedure that tests hypotheses top-down through a tree: test the overall null at the root, then groups of blocks, then individual blocks, stopping any branch where the null is not rejected. In the same 44-block design, this approach detects effects in 44\% of non-null blocks -- roughly four times the detection rate. A stopping rule and valid tests at each node suffice for weak FWER control. We show that the strong-sense FWER depends on how rejection probabilities accumulate along paths through the tree. This yields a diagnostic: when power decays fast enough relative to branching, no adjustment is needed; otherwise, an adaptive $\alpha$-adjustment restores control. We apply the method to 25 MDRC education trials and provide an R package, \texttt{manytestsr}.