Invariance Pair-Guided Learning: Enhancing Robustness in Neural Networks

TL;DR

This paper introduces Invariance Pair-Guided Learning, a novel training technique that improves neural network robustness to distribution shifts by leveraging input pairs to enforce invariance, without requiring multiple training domains or group labels.

Contribution

The method provides a new way to guide neural networks using invariance pairs, enhancing out-of-distribution generalization without additional data or complex preprocessing.

Findings

Effective on ColoredMNIST, Waterbird-100, CelebA datasets

Demonstrates robustness to group shifts

Outperforms existing approaches in generalization

Abstract

Out-of-distribution generalization of machine learning models remains challenging since the models are inherently bound to the training data distribution. This especially manifests, when the learned models rely on spurious correlations. Most of the existing approaches apply data manipulation, representation learning, or learning strategies to achieve generalizable models. Unfortunately, these approaches usually require multiple training domains, group labels, specialized augmentation, or pre-processing to reach generalizable models. We propose a novel approach that addresses these limitations by providing a technique to guide the neural network through the training phase. We first establish input pairs, representing the spurious attribute and describing the invariance, a characteristic that should not affect the outcome of the model. Based on these pairs, we form a corrective gradient complementing the traditional gradient descent approach. We further make this correction mechanism adaptive based on a predefined invariance condition. Experiments on ColoredMNIST, Waterbird-100, and CelebA datasets demonstrate the effectiveness of our approach and the robustness to group shifts.