Does Residuals-on-Residuals Regression Produce Representative Estimates of Causal Effects?

TL;DR

This paper critically examines the residuals-on-residuals regression (RORR) estimator used in causal inference, revealing its limitations with heterogeneous effects and proposing an alternative estimator suitable for large datasets, demonstrated through simulations and real-world Netflix data.

Contribution

The paper identifies the limitations of RORR in estimating causal effects with heterogeneous treatments and introduces a new estimator that accurately targets the average causal derivative in large datasets.

Findings

RORR converges to a conditional variance-weighted derivative, not the ACD.

RORR may not reflect the true average treatment effect in heterogeneous settings.

The proposed estimator performs better in large-scale observational data.

Abstract

Double Machine Learning is widely used to estimate causal treatment effects in large-scale observational data. The ``residuals-on-residuals'' regression estimator (RORR) is especially popular for its simplicity and computational tractability. However, when treatment effects are heterogeneous, the proper interpretation of RORR may not be widely understood. We show that for many-valued treatments with continuous dose-response functions, RORR converges to a conditional variance-weighted average of derivatives evaluated at points not in the observed dataset. This estimand does not in general equal the Average Causal Derivative (ACD). Hence, even if all units share the same dose-response function, RORR may not converge to an average treatment effect in the population represented by the sample. We propose an alternative estimator for the ACD that is suitable for large datasets. We demonstrate the pitfalls of RORR and the favorable properties of the proposed estimator through an illustrative numerical example and with real-world data from Netflix. Our methodology is deployed in Netflix's internal observational causal inference platform, where it regularly powers causal research and decision-making at scale.