Neyman Meets Causal Machine Learning: Experimental Evaluation of Individualized Treatment Rules

TL;DR

This paper extends Neyman's classical experimental framework to evaluate individualized treatment rules derived from causal machine learning, accounting for training uncertainty and demonstrating efficiency advantages over traditional methods.

Contribution

It introduces a Neyman-based methodology for experimental evaluation of ITRs that is applicable regardless of the machine learning algorithms used, incorporating cross-fitting uncertainty.

Findings

Neyman's approach can evaluate ITRs regardless of ML properties.

Ex-post evaluation can be more efficient than ex-ante randomization.

Neyman's framework remains relevant for modern causal inference.

Abstract

A century ago, Neyman showed how to evaluate the efficacy of treatment using a randomized experiment under a minimal set of assumptions. This classical repeated sampling framework serves as a basis of routine experimental analyses conducted by today's scientists across disciplines. In this paper, we demonstrate that Neyman's methodology can also be used to experimentally evaluate the efficacy of individualized treatment rules (ITRs), which are derived by modern causal machine learning algorithms. In particular, we show how to account for additional uncertainty resulting from a training process based on cross-fitting. The primary advantage of Neyman's approach is that it can be applied to any ITR regardless of the properties of machine learning algorithms that are used to derive the ITR. We also show, somewhat surprisingly, that for certain metrics, it is more efficient to conduct this ex-post experimental evaluation of an ITR than to conduct an ex-ante experimental evaluation that randomly assigns some units to the ITR. Our analysis demonstrates that Neyman's repeated sampling framework is as relevant for causal inference today as it has been since its inception.