Policy Learning with Rare Outcomes

TL;DR

This paper evaluates the performance of various policy learning algorithms using synthetic data and a real-world case study, highlighting the effectiveness of CATE-based policy trees and specific machine learning methods.

Contribution

It provides a comparative analysis of policy learning algorithms' performance in realistic scenarios and demonstrates their application to health policy in Indonesia.

Findings

Policy trees based on estimated CATEs outperform those from doubly-robust scores.

Causal Forests and Normalised Double-Robust Learner perform consistently well.

Bayesian Additive Regression Trees perform poorly in the evaluated settings.

Abstract

Machine learning (ML) estimates of conditional average treatment effects (CATE) can guide policy decisions, either by allowing targeting of individuals with beneficial CATE estimates, or as inputs to decision trees that optimise overall outcomes. There is limited information available regarding how well these algorithms perform in real-world policy evaluation scenarios. Using synthetic data, we compare the finite sample performance of different policy learning algorithms, machine learning techniques employed during their learning phases, and methods for presenting estimated policy values. For each algorithm, we assess the resulting treatment allocation by measuring deviation from the ideal ("oracle") policy. Our main finding is that policy trees based on estimated CATEs outperform trees learned from doubly-robust scores. Across settings, Causal Forests and the Normalised Double-Robust Learner perform consistently well, while Bayesian Additive Regression Trees perform poorly. These methods are then applied to a case study targeting optimal allocation of subsidised health insurance, with the goal of reducing infant mortality in Indonesia.