An Aggregation Scheme for Increased Power

TL;DR

This paper introduces PWRD, a novel aggregation scheme that leverages a theory of change to enhance statistical power in randomized trials with longitudinal data, especially when effects are delayed and non-uniform.

Contribution

The paper proposes PWRD, a power-maximizing weighting method that converts a theory of change into a more efficient test statistic for repeated measurements.

Findings

PWRD improves power comparable to doubling the number of clusters.

Applied to a reading intervention, PWRD increased test power.

Method enhances efficiency in trials with delayed and non-uniform effects.

Abstract

We present an aggregation scheme that increases power in randomized controlled trials and quasi-experiments when the intervention possesses a robust and well-articulated theory of change. Longitudinal data analyzing interventions often include multiple observations on individuals, some of which may be more likely to manifest a treatment effect than others. An intervention's theory of change provides guidance as to which of those observations are best situated to exhibit that treatment effect. Our power-maximizing weighting for repeated-measurements with delayed-effects scheme, PWRD aggregation, converts the theory of change into a test statistic with improved asymptotic relative efficiency, delivering tests with greater statistical power. We illustrate this method on an IES-funded cluster randomized trial testing the efficacy of a reading intervention designed to assist early elementary students at risk of falling behind their peers. The salient theory of change holds program benefits to be delayed and non-uniform, experienced after a student's performance stalls. In this instance, the PWRD technique's effect on power is found to be comparable to that of doubling the number of clusters in the experiment.