Environment Invariant Linear Least Squares

TL;DR

This paper introduces a novel environment invariant linear least squares (EILLS) method for multi-environment linear regression that leverages invariance across environments to accurately estimate true parameters and perform variable selection.

Contribution

It proposes the first statistically efficient invariance learning approach in the general linear model, applicable without structural assumptions, and provides non-asymptotic error bounds and variable selection consistency.

Findings

EILLS achieves near-minimal identification conditions.

The method provides non-asymptotic $\,\ell_2$ error bounds.

The $\,\ell_0$ penalized EILLS attains variable selection consistency.

Abstract

This paper considers a multi-environment linear regression model in which data from multiple experimental settings are collected. The joint distribution of the response variable and covariates may vary across different environments, yet the conditional expectations of $y$ given the unknown set of important variables are invariant. Such a statistical model is related to the problem of endogeneity, causal inference, and transfer learning. The motivation behind it is illustrated by how the goals of prediction and attribution are inherent in estimating the true parameter and the important variable set. We construct a novel environment invariant linear least squares (EILLS) objective function, a multi-environment version of linear least-squares regression that leverages the above conditional expectation invariance structure and heterogeneity among different environments to determine the true parameter. Our proposed method is applicable without any additional structural knowledge and can identify the true parameter under a near-minimal identification condition. We establish non-asymptotic $\ell_2$ error bounds on the estimation error for the EILLS estimator in the presence of spurious variables. Moreover, we further show that the $\ell_0$ penalized EILLS estimator can achieve variable selection consistency in high-dimensional regimes. These non-asymptotic results demonstrate the sample efficiency of the EILLS estimator and its capability to circumvent the curse of endogeneity in an algorithmic manner without any prior structural knowledge. To the best of our knowledge, this paper is the first to realize statistically efficient invariance learning in the general linear model.