DivIL: Unveiling and Addressing Over-Invariance for Out-of- Distribution Generalization

TL;DR

This paper identifies the problem of over-invariance in invariant learning methods for out-of-distribution generalization, and proposes DivIL, a contrastive learning-based approach to mitigate this issue, demonstrating improved performance across multiple datasets and models.

Contribution

The paper introduces the concept of over-invariance in invariant learning and proposes DivIL, a novel method combining contrastive learning and masking to address it.

Findings

Over-invariance occurs in classic IL methods, degrading generalization.

DivIL effectively alleviates over-invariance, improving OOD performance.

Experimental results confirm DivIL's effectiveness across diverse datasets and models.

Abstract

Out-of-distribution generalization is a common problem that expects the model to perform well in the different distributions even far from the train data. A popular approach to addressing this issue is invariant learning (IL), in which the model is compiled to focus on invariant features instead of spurious features by adding strong constraints during training. However, there are some potential pitfalls of strong invariant constraints. Due to the limited number of diverse environments and over-regularization in the feature space, it may lead to a loss of important details in the invariant features while alleviating the spurious correlations, namely the over-invariance, which can also degrade the generalization performance. We theoretically define the over-invariance and observe that this issue occurs in various classic IL methods. To alleviate this issue, we propose a simple approach Diverse Invariant Learning (DivIL) by adding the unsupervised contrastive learning and the random masking mechanism compensatory for the invariant constraints, which can be applied to various IL methods. Furthermore, we conduct experiments across multiple modalities across 12 datasets and 6 classic models, verifying our over-invariance insight and the effectiveness of our DivIL framework. Our code is available at https://github.com/kokolerk/DivIL.