Randomization Bias in Field Trials to Evaluate Targeting Methods

TL;DR

This paper examines bias in field trial evaluations of targeting methods, demonstrating that randomized designs can be biased while survey-based approaches are unbiased, with implications for policy evaluation.

Contribution

It identifies bias in randomized controlled trials for targeting methods and proposes survey-based design as an unbiased alternative, validated through simulations.

Findings

Randomized controlled trials can produce biased estimates of targeting method effectiveness.

Survey-based evaluation methods are unbiased in measuring targeting performance.

Application to Chicago's lead hazard policy demonstrated the practical advantage of survey design.

Abstract

This paper studies the evaluation of methods for targeting the allocation of limited resources to a high-risk subpopulation. We consider a randomized controlled trial to measure the difference in efficiency between two targeting methods and show that it is biased. An alternative, survey-based design is shown to be unbiased. Both designs are simulated for the evaluation of a policy to target lead hazard investigations using a predictive model. Based on our findings, we advised the Chicago Department of Public Health to use the survey design for their field trial. Our work anticipates further developments in economics that will be important as predictive modeling becomes an increasingly common policy tool.