Treatment effect: a critique

TL;DR

This paper critically examines two main perspectives on treatment effects in statistics, contrasting model-based and counterfactual approaches, and discusses their implications for scientific inference and generalization.

Contribution

It clarifies the relationship between Fisherian inference and counterfactual frameworks, highlighting concerns about model-free definitions as inference targets.

Findings

Contrasts between model-based and counterfactual treatment effects

Highlights limitations of model-free definitions for scientific inference

Discusses implications for generalizing treatment effect conclusions

Abstract

Two broad positions within statistics define a treatment effect, on the one hand, as a parameter of a statistical model, and on the other, as an appropriate population-level difference in outcomes or counterfactual outcomes under the different treatment regimes. This short expository paper presents some simple but consequential insights on the two formulations, contrasting the answers under the most favourable fictitious idealisation for the counterfactual framework. These observations clarify the relationship between Fisherian model-based inference and modern counterfactual formulations, and emphasise concerns, raised by Cox and others, regarding the suitability of model-free definitions as targets of inference when scientific conclusions are intended to generalise beyond the observed sample. Parts of the paper are necessarily controversial; we follow Cox (1958a) in not putting these forward in any dogmatic spirit.