When Invariant Representation Learning Meets Label Shift: Insufficiency and Theoretical Insights

TL;DR

This paper investigates the limitations of invariant representation learning under label shift, introduces theoretical bounds and a kernel embedding correction algorithm, demonstrating the importance of generalized label shift correction for robust generalization.

Contribution

It provides new theoretical insights into GLS, proves the insufficiency of invariant learning alone, and proposes a novel KECA algorithm for effective shift correction.

Findings

Invariant representation learning is insufficient under GLS.

GLS correction is necessary and sufficient for better generalization.

The proposed KECA algorithm outperforms existing methods in experiments.

Abstract

As a crucial step toward real-world learning scenarios with changing environments, dataset shift theory and invariant representation learning algorithm have been extensively studied to relax the identical distribution assumption in classical learning setting. Among the different assumptions on the essential of shifting distributions, generalized label shift (GLS) is the latest developed one which shows great potential to deal with the complex factors within the shift. In this paper, we aim to explore the limitations of current dataset shift theory and algorithm, and further provide new insights by presenting a comprehensive understanding of GLS. From theoretical aspect, two informative generalization bounds are derived, and the GLS learner is proved to be sufficiently close to optimal target model from the Bayesian perspective. The main results show the insufficiency of invariant representation learning, and prove the sufficiency and necessity of GLS correction for generalization, which provide theoretical supports and innovations for exploring generalizable model under dataset shift. From methodological aspect, we provide a unified view of existing shift correction frameworks, and propose a kernel embedding-based correction algorithm (KECA) to minimize the generalization error and achieve successful knowledge transfer. Both theoretical results and extensive experiment evaluations demonstrate the sufficiency and necessity of GLS correction for addressing dataset shift and the superiority of proposed algorithm.