Does Regression Produce Representative Causal Rankings?

TL;DR

This paper investigates how linear regression and the Partially Linear Model can produce inconsistent treatment effect rankings due to ranking reversals caused by treatment effect heterogeneity and overlap-weighting.

Contribution

It identifies conditions leading to ranking reversals in treatment effect estimates and provides a formal characterization of when these inconsistencies occur.

Findings

Overlap-weighted estimates can reverse true treatment rankings.

Ranking reversals depend on specific conditions in the PLM.

Simulation studies illustrate when and how reversals happen.

Abstract

We examine the challenges in ranking multiple treatments based on their estimated effects when using linear regression or its popular double-machine-learning variant, the Partially Linear Model (PLM), in the presence of treatment effect heterogeneity. We demonstrate by example that overlap-weighting performed by linear models like PLM can produce Weighted Average Treatment Effects (WATE) that have rankings that are inconsistent with the rankings of the underlying Average Treatment Effects (ATE). We define this as ranking reversals and derive a necessary and sufficient condition for ranking reversals under the PLM. We conclude with several simulation studies conditions under which ranking reversals occur.