Bayesian Invariance Modeling of Multi-Environment Data

TL;DR

This paper introduces Bayesian Invariant Prediction (BIP), a probabilistic framework for identifying invariant features across multiple environments, improving accuracy and scalability over previous methods.

Contribution

The paper develops BIP, a Bayesian model for invariant prediction, with a variational approximation VI-BIP, demonstrating theoretical consistency and practical advantages.

Findings

BIP accurately identifies invariant features in simulations.

VI-BIP scales efficiently to many features.

BIP outperforms existing methods in real data applications.

Abstract

Invariant prediction [Peters et al., 2016] analyzes feature/outcome data from multiple environments to identify invariant features - those with a stable predictive relationship to the outcome. Such features support generalization to new environments and help reveal causal mechanisms. Previous methods have primarily tackled this problem through hypothesis testing or regularized optimization. Here we develop Bayesian Invariant Prediction (BIP), a probabilistic model for invariant prediction. BIP encodes the indices of invariant features as a latent variable and recover them by posterior inference. Under the assumptions of Peters et al. [2016], the BIP posterior targets the true invariant features. We prove that the posterior is consistent and that greater environment heterogeneity leads to faster posterior contraction. To handle many features, we design an efficient variational approximation called VI-BIP. In simulations and real data, we find that BIP and VI-BIP are more accurate and scalable than existing methods for invariant prediction.