Interpolation and Regularization for Causal Learning

TL;DR

This paper investigates whether interpolating estimators can effectively learn causal models from observational data, revealing that under certain assumptions, stronger regularization is necessary for causal learning compared to statistical learning.

Contribution

It provides a theoretical analysis of the causal risk of interpolators and ridge regressors in high-dimensional causal models, resolving a recent conjecture and introducing confounding strength as a key measure.

Findings

Interpolators are not optimal under the independent causal mechanisms assumption.

Stronger regularization is required for causal learning than for statistical learning.

Negative confounding strength can make interpolators optimal and even suggest negative regularization.

Abstract

We study the problem of learning causal models from observational data through the lens of interpolation and its counterpart -- regularization. A large volume of recent theoretical, as well as empirical work, suggests that, in highly complex model classes, interpolating estimators can have good statistical generalization properties and can even be optimal for statistical learning. Motivated by an analogy between statistical and causal learning recently highlighted by Janzing (2019), we investigate whether interpolating estimators can also learn good causal models. To this end, we consider a simple linearly confounded model and derive precise asymptotics for the *causal risk* of the min-norm interpolator and ridge-regularized regressors in the high-dimensional regime. Under the principle of independent causal mechanisms, a standard assumption in causal learning, we find that interpolators cannot be optimal and causal learning requires stronger regularization than statistical learning. This resolves a recent conjecture in Janzing (2019). Beyond this assumption, we find a larger range of behavior that can be precisely characterized with a new measure of *confounding strength*. If the confounding strength is negative, causal learning requires weaker regularization than statistical learning, interpolators can be optimal, and the optimal regularization can even be negative. If the confounding strength is large, the optimal regularization is infinite, and learning from observational data is actively harmful.