Propensity score adjustment when errors in achievement measures inform treatment assignment

TL;DR

This paper develops propensity score methods that account for measurement error in achievement data to better evaluate educational interventions targeting achievement gaps.

Contribution

It introduces propensity score adjustments that incorporate measurement error, improving bias reduction and overlap in observational studies of educational outcomes.

Findings

Improved bias reduction in treatment effect estimates.

Enhanced overlap in propensity score matching.

Validated methods through simulation and real data application.

Abstract

U.S. state education agencies mark schools displaying achievement gaps between demographic subgroups as needing improvement. Some schools may have few students in these subgroups, such that average end-of-year test scores only noisily measure the average "true" score--the score one would expect if students took the test many times. This, in addition to the masking of small subgroup averages in publicly available assessment data, poses challenges for evaluating interventions aimed at closing achievement gaps. We introduce propensity score estimates designed to achieve balance on subgroup average true scores. These estimates are available even when noisy measurements are not and improve overlap compared to those that ignore measurement error, leading to greater bias reduction of matching estimators. We demonstrate our methods through simulation and an application to a statewide initiative in Texas for curbing summer learning loss.