On Invariance Penalties for Risk Minimization

TL;DR

This paper critiques the original invariance penalty in IRM, proposes a new penalty based on the Gramian matrix, and demonstrates improved invariance recovery in linear settings through experiments.

Contribution

It introduces an alternative invariance penalty using the Gramian matrix, addressing limitations of the original IRM approach.

Findings

The new penalty recovers invariant representations in linear models.

It outperforms the original IRM penalty on domain generalization benchmarks.

The approach is effective under mild non-degeneracy conditions.

Abstract

The Invariant Risk Minimization (IRM) principle was first proposed by Arjovsky et al. [2019] to address the domain generalization problem by leveraging data heterogeneity from differing experimental conditions. Specifically, IRM seeks to find a data representation under which an optimal classifier remains invariant across all domains. Despite the conceptual appeal of IRM, the effectiveness of the originally proposed invariance penalty has recently been brought into question. In particular, there exists counterexamples for which that invariance penalty can be arbitrarily small for non-invariant data representations. We propose an alternative invariance penalty by revisiting the Gramian matrix of the data representation. We discuss the role of its eigenvalues in the relationship between the risk and the invariance penalty, and demonstrate that it is ill-conditioned for said counterexamples. The proposed approach is guaranteed to recover an invariant representation for linear settings under mild non-degeneracy conditions. Its effectiveness is substantiated by experiments on DomainBed and InvarianceUnitTest, two extensive test beds for domain generalization.