Distribution Shift Is Key to Learning Invariant Prediction

TL;DR

This paper reveals that significant distribution shifts across training domains can enhance ERM performance and facilitate learning invariant predictions, challenging the assumption that distribution shift is always detrimental.

Contribution

The study provides theoretical bounds and empirical evidence showing distribution shift can improve ERM and aid in learning models that are invariant across domains.

Findings

Large distribution shifts can improve ERM performance.

Distribution shift influences the ability to learn invariant predictions.

Empirical results show models approximate Oracle predictions with increased shift.

Abstract

An interesting phenomenon arises: Empirical Risk Minimization (ERM) sometimes outperforms methods specifically designed for out-of-distribution tasks. This motivates an investigation into the reasons behind such behavior beyond algorithmic design. In this study, we find that one such reason lies in the distribution shift across training domains. A large degree of distribution shift can lead to better performance even under ERM. Specifically, we derive several theoretical and empirical findings demonstrating that distribution shift plays a crucial role in model learning and benefits learning invariant prediction. Firstly, the proposed upper bounds indicate that the degree of distribution shift directly affects the prediction ability of the learned models. If it is large, the models' ability can increase, approximating invariant prediction models that make stable predictions under arbitrary known or unseen domains; and vice versa. We also prove that, under certain data conditions, ERM solutions can achieve performance comparable to that of invariant prediction models. Secondly, the empirical validation results demonstrated that the predictions of learned models approximate those of Oracle or Optimal models, provided that the degree of distribution shift in the training data increases.