Better Measurement or Larger Samples? Data Collection for Policy Learning with Unobserved Heterogeneity

TL;DR

This paper analyzes how data collection strategies, including sample size and measurement precision, impact policy effectiveness when unobserved heterogeneity affects treatment responses.

Contribution

It introduces a framework for designing optimal data collection plans balancing measurement precision and sample size to improve policy outcomes.

Findings

Including a proxy for entrepreneurs' skills increases welfare by 5%.

Halves the probability of welfare losses with better proxies.

Provides a method to allocate resources between measurement precision and sample size.

Abstract

Empirical research shows that individuals' responses to treatments vary along latent characteristics, such as innate ability or motivation. Therefore, a policymaker seeking to maximize welfare may consider designing policies based on observed characteristics and estimated latent traits. I characterize how the estimates' precision affects the worst-case performance of policies deriving rate-sharp regret bounds for assignment rules that include or exclude them, highlighting new trade-offs with the policy space complexity. I then study how a policymaker can solve such trade-offs by designing tailored data collections and derive a sufficient condition for a collection plan to be minimax optimal. In an empirical application in development economics, I show that including a proxy for entrepreneurs' business skills in targeting cash transfers increases welfare by 5%, and halves the probability of generating welfare losses. Moreover, I estimate the optimal allocation of resources between improving the precision of the proxy via repeated measurements, and increasing sample size.