Counterfactual (Non-)identifiability of Learned Structural Causal Models

TL;DR

This paper investigates the limits of counterfactual inference in learned structural causal models, revealing non-identifiability issues and proposing a method to estimate worst-case errors to assess model reliability.

Contribution

It demonstrates non-identifiability of counterfactuals in general DSCMs, proves identifiability under certain conditions, and introduces a practical error estimation approach for counterfactual predictions.

Findings

Counterfactuals are non-identifiable in general DSCMs without assumptions.

Identifiability is proven for monotonic mechanisms with single-dimensional exogenous variables.

The proposed error estimation method effectively assesses counterfactual prediction reliability.

Abstract

Recent advances in probabilistic generative modeling have motivated learning Structural Causal Models (SCM) from observational datasets using deep conditional generative models, also known as Deep Structural Causal Models (DSCM). If successful, DSCMs can be utilized for causal estimation tasks, e.g., for answering counterfactual queries. In this work, we warn practitioners about non-identifiability of counterfactual inference from observational data, even in the absence of unobserved confounding and assuming known causal structure. We prove counterfactual identifiability of monotonic generation mechanisms with single dimensional exogenous variables. For general generation mechanisms with multi-dimensional exogenous variables, we provide an impossibility result for counterfactual identifiability, motivating the need for parametric assumptions. As a practical approach, we propose a method for estimating worst-case errors of learned DSCMs' counterfactual predictions. The size of this error can be an essential metric for deciding whether or not DSCMs are a viable approach for counterfactual inference in a specific problem setting. In evaluation, our method confirms negligible counterfactual errors for an identifiable SCM from prior work, and also provides informative error bounds on counterfactual errors for a non-identifiable synthetic SCM.