Generalizability with ignorance in mind: learning what we do (not) know for archetypes discovery

TL;DR

This paper introduces a framework for identifying when treatment effects are generalizable across different environments and when researchers should acknowledge ignorance and gather more data, enhancing policy analysis.

Contribution

It develops a decision-theoretic approach to partition observations based on the stability of treatment effects, with finite-sample guarantees and asymptotic inference.

Findings

Framework successfully identifies generalizable effects across environments

Reanalysis of anti-poverty program demonstrates practical utility

Provides finite-sample regret bounds and inference methods

Abstract

When studying policy interventions, researchers often pursue two goals: i) identifying for whom the program has the largest effects (heterogeneity) and ii) determining whether those patterns of treatment effects have predictive power across environments (generalizability). We develop a framework to learn when and how to partition observations into groups of individual and environmental characterstics within which treatment effects are predictively stable, and when instead extrapolation is unwarranted and further evidence is needed. Our procedure determines in which contexts effects are generalizable and when, instead, researchers should admit ignorance and collect more data. We provide a decision-theoretic foundation, derive finite-sample regret guarantees, and establish asymptotic inference results. We illustrate the benefits of our approach by reanalyzing a multifaceted anti-poverty program across six countries.