Causal Regularization: On the trade-off between in-sample risk and out-of-sample risk guarantees

TL;DR

This paper introduces causal regularization to balance in-sample risk minimization and out-of-sample risk guarantees, providing a method to select models that optimize predictive stability across different data environments.

Contribution

It proposes a novel causal regularization approach that balances in-sample and out-of-sample risks and demonstrates how to select optimal regularization via cross-validation.

Findings

Higher regularization improves risk stability in population models.

Finite data reduces the ability to identify models with guaranteed out-of-sample risk.

Cross-validation can effectively select the optimal causal regularizer in empirical settings.

Abstract

Invariant prediction uses the prediction stability of causal relationships across different environments to identify causal variables. Conversely, using causal variables gives prediction guarantees even in out-of-sample data settings. In this paper, we investigate the identification of causal-like models from in-sample data that ensure out-of-sample risk guarantees when predicting a target variable from an arbitrary set of covariates. Ordinary least squares minimizes in-sample risk but offers limited out-of-sample guarantees, while causal models optimize out-of-sample guarantees at the expense of in-sample performance. We introduce a form of \textit{causal regularization} to balance these properties. In the population setting, higher regularization yields estimators with greater risk stability, albeit with increased in-sample risk. Empirically, however, there is a further trade-off to consider, as finite in-sample data reduced the ability to correctly identify models with high out-of-sample risk guarantees. We show how in such empirical settings the optimal causal regularizer can be found via cross-validation.