Invariant Correlation of Representation with Label: Enhancing Domain Generalization in Noisy Environments

TL;DR

This paper introduces ICorr, a new method that improves domain generalization in noisy environments by ensuring invariant correlation between representations and labels, supported by theoretical analysis and empirical validation.

Contribution

The paper proposes ICorr, a novel approach that maintains invariant correlation in noisy settings, addressing limitations of existing IRM-based methods through causality-based insights.

Findings

ICorr outperforms existing methods on noisy datasets

Invariant correlation is necessary for optimal invariant predictors in noisy environments

Theoretical analysis explains why ICorr succeeds where others fail

Abstract

The Invariant Risk Minimization (IRM) approach aims to address the challenge of domain generalization by training a feature representation that remains invariant across multiple environments. However, in noisy environments, IRM-related techniques such as IRMv1 and VREx may be unable to achieve the optimal IRM solution, primarily due to erroneous optimization directions. To address this issue, we introduce ICorr (an abbreviation for Invariant Correlation), a novel approach designed to surmount the above challenge in noisy settings. Additionally, we dig into a case study to analyze why previous methods may lose ground while ICorr can succeed. Through a theoretical lens, particularly from a causality perspective, we illustrate that the invariant correlation of representation with label is a necessary condition for the optimal invariant predictor in noisy environments, whereas the optimization motivations for other methods may not be. Furthermore, we empirically demonstrate the effectiveness of ICorr by comparing it with other domain generalization methods on various noisy datasets. The code is available at https://github.com/Alexkael/ICorr.