Continuous Invariance Learning

TL;DR

This paper introduces Continuous Invariance Learning (CIL), a novel adversarial approach that effectively learns invariant features across continuous domains, addressing limitations of existing methods that discretize such domains.

Contribution

The paper proposes CIL, a new adversarial method for continuous invariance learning, with theoretical analysis and empirical validation showing its advantages over existing techniques.

Findings

CIL outperforms baseline methods on synthetic datasets.

CIL achieves better generalization on real-world production data.

Theoretical analysis confirms CIL's superiority over existing invariance methods.

Abstract

Invariance learning methods aim to learn invariant features in the hope that they generalize under distributional shifts. Although many tasks are naturally characterized by continuous domains, current invariance learning techniques generally assume categorically indexed domains. For example, auto-scaling in cloud computing often needs a CPU utilization prediction model that generalizes across different times (e.g., time of a day and date of a year), where `time' is a continuous domain index. In this paper, we start by theoretically showing that existing invariance learning methods can fail for continuous domain problems. Specifically, the naive solution of splitting continuous domains into discrete ones ignores the underlying relationship among domains, and therefore potentially leads to suboptimal performance. To address this challenge, we then propose Continuous Invariance Learning (CIL), which extracts invariant features across continuously indexed domains. CIL is a novel adversarial procedure that measures and controls the conditional independence between the labels and continuous domain indices given the extracted features. Our theoretical analysis demonstrates the superiority of CIL over existing invariance learning methods. Empirical results on both synthetic and real-world datasets (including data collected from production systems) show that CIL consistently outperforms strong baselines among all the tasks.