Prediction decomposition for causal analysis

TL;DR

This paper introduces a decomposition of ML prediction into components relevant for causal analysis, proposing a new metric to improve model selection and treatment effect estimation from panel data.

Contribution

It presents a novel decomposition of predictions into three components, highlighting the importance of within-unit-across-time accuracy for causal inference, and proposes a diagnostic metric for model selection.

Findings

Counterfactual-treatment-effect component is key for true treatment effect recovery.

Within-unit-across-time prediction accuracy correlates better with causal effect than overall accuracy.

The proposed metric enables model diagnostics and unbiased treatment effect estimation under certain assumptions.

Abstract

There is rising interest in using Machine Learning (ML) model predictions as outcomes in causal analysis. However, these methods have faced challenges in finding the true treatment effects. It is also challenging to make choices about which prediction models to choose, since we are interested not only in the accuracy of the prediction but in its ability to produce the correct causal effect in the analysis. In this paper I propose a decomposition of the prediction into between-unit prediction ($\eta_\mu$), within-unit-across-time prediction ($\eta_\epsilon$), and counterfactual-treatment-effect prediction ($\eta_T$). I show that the counterfactual-treatment-effect component is the one that determines whether the model recovers the true treatment effect, but only the first two components can be estimated from non-experimental data. I argue that within-unit-across-time prediction accuracy ($\eta_\epsilon$) is a structurally better proxy for the counterfactual-treatment-effect component ($\eta_T$) than overall prediction accuracy, and propose a metric to estimate it from panel data with at least two time periods. This metric serves as a diagnostic and model-selection tool for choosing ML models for causal analysis. Under the stronger assumption that $\eta_T \approx \eta_\epsilon$, it also enables constructing an approximately unbiased estimate of the treatment effect. I develop the theoretical framework and illustrate it with simulations of synthetic data.