Comparing Targeting Strategies for Maximizing Social Welfare with Limited Resources

TL;DR

This paper empirically compares different machine learning-based targeting strategies for social interventions, demonstrating that treatment effect-based targeting can outperform risk-based methods when accurate effect estimates are available, but current data limitations hinder its practical application.

Contribution

It provides the first empirical evaluation of targeting strategies using real RCT data across multiple social domains, highlighting the potential and challenges of treatment effect-based targeting.

Findings

Treatment effect targeting outperforms risk-based targeting with accurate effect estimates.

Current data limitations prevent reliable estimation of heterogeneous treatment effects.

Improving data collection and modeling can unlock the benefits of treatment effect-based targeting.

Abstract

Machine learning is increasingly used to select which individuals receive limited-resource interventions in domains such as human services, education, development, and more. However, it is often not apparent what the right quantity is for models to predict. Policymakers rarely have access to data from a randomized controlled trial (RCT) that would enable accurate estimates of which individuals would benefit more from the intervention, while observational data creates a substantial risk of bias in treatment effect estimates. Practitioners instead commonly use a technique termed ``risk-based targeting" where the model is just used to predict each individual's status quo outcome (an easier, non-causal task). Those with higher predicted risk are offered treatment. There is currently almost no empirical evidence to inform which choices lead to the most effective machine learning-informed targeting strategies in social domains. In this work, we use data from 5 real-world RCTs in a variety of domains to empirically assess such choices. We find that when treatment effects can be estimated with high accuracy (which we simulate by allowing the model to partially observe outcomes in advance), treatment effect based targeting substantially outperforms risk-based targeting, even when treatment effect estimates are biased. Moreover, these results hold even when the policymaker has strong normative preferences for assisting higher-risk individuals. However, the features and data actually available in most RCTs we examine do not suffice for accurate estimates of heterogeneous treatment effects. Our results suggest treatment effect targeting has significant potential benefits, but realizing these benefits requires improvements to data collection and model training beyond what is currently common in practice.