Local Randomization Regression Discontinuity Designs when Test Scores are the Running Variable

TL;DR

This paper investigates the validity of local randomization methods in regression discontinuity designs when test scores are used as the running variable, revealing potential biases and limitations in such scenarios.

Contribution

It derives a bias formula for local randomization RDD estimates with test scores and highlights issues with applying these methods to human-developed measures.

Findings

Bias in LATE estimate is proportional to test reliability and score difference

Local randomization methods may be invalid for test score running variables

Potential for significant bias when using test scores in RDDs

Abstract

Explanations of the internal validity of regression discontinuity designs (RDD) generally appeal to the idea that RDDs are ``as good as" random near the treatment cut point. Cattaneo, Frandsen, and Titiunik (2015) are the first to take this justification to its full conclusion and propose estimating the RDD local average treatment effect (LATE) the same as one would a randomized experiment. This paper explores the implications of analyzing an RDD as a local random experiment when the running variable is a test score. I derive a formula for the bias in the LATE estimate estimated using the local randomization method, $a\rho\Delta$. Where $a$ is the relationship between latent proficiency and the potential outcome absent treatment, $\rho$ is the test reliability, and $\Delta$ is the distance between the treatment and control running variable value. I use this quantification of the bias to demonstrate that local randomization and related design based methods for estimating RDDs are problematic when the running variable is test score or other human developed measure (e.g., medical tests).