Invariance Learning based on Label Hierarchy

TL;DR

This paper introduces a novel invariance learning framework that leverages label hierarchy to train invariant predictors using data from a single domain, reducing annotation costs and improving generalization across domains.

Contribution

It proposes a new invariance learning method based on label hierarchy and cross-validation techniques for hyperparameter tuning, addressing limitations of existing multi-domain approaches.

Findings

Effective invariant predictor estimation from single-domain data.

Successful hyperparameter selection via proposed cross-validation methods.

Empirical validation demonstrating improved domain generalization.

Abstract

Deep Neural Networks inherit spurious correlations embedded in training data and hence may fail to predict desired labels on unseen domains (or environments), which have different distributions from the domain used in training. Invariance Learning (IL) has been developed recently to overcome this shortcoming; using training data in many domains, IL estimates such a predictor that is invariant to a change of domain. However, the requirement of training data in multiple domains is a strong restriction of IL, since it often needs high annotation cost. We propose a novel IL framework to overcome this problem. Assuming the availability of data from multiple domains for a higher level of classification task, for which the labeling cost is low, we estimate an invariant predictor for the target classification task with training data in a single domain. Additionally, we propose two cross-validation methods for selecting hyperparameters of invariance regularization to solve the issue of hyperparameter selection, which has not been handled properly in existing IL methods. The effectiveness of the proposed framework, including the cross-validation, is demonstrated empirically, and the correctness of the hyperparameter selection is proved under some conditions.