Prominent Roles of Conditionally Invariant Components in Domain Adaptation: Theory and Algorithms

TL;DR

This paper investigates the role of conditionally invariant components (CICs) in domain adaptation, proposing new algorithms and theoretical guarantees that improve performance across various datasets by leveraging CICs.

Contribution

It introduces the importance-weighted conditional invariant penalty (IW-CIP) algorithm and demonstrates how CICs enhance domain adaptation methods with theoretical guarantees.

Findings

IW-CIP provides target risk guarantees beyond covariate and label shift.

CICs help identify discrepancies between source and target risks.

Incorporating CICs improves domain invariant projection (DIP) performance.

Abstract

Domain adaptation (DA) is a statistical learning problem that arises when the distribution of the source data used to train a model differs from that of the target data used to evaluate the model. While many DA algorithms have demonstrated considerable empirical success, blindly applying these algorithms can often lead to worse performance on new datasets. To address this, it is crucial to clarify the assumptions under which a DA algorithm has good target performance. In this work, we focus on the assumption of the presence of conditionally invariant components (CICs), which are relevant for prediction and remain conditionally invariant across the source and target data. We demonstrate that CICs, which can be estimated through conditional invariant penalty (CIP), play three prominent roles in providing target risk guarantees in DA. First, we propose a new algorithm based on CICs, importance-weighted conditional invariant penalty (IW-CIP), which has target risk guarantees beyond simple settings such as covariate shift and label shift. Second, we show that CICs help identify large discrepancies between source and target risks of other DA algorithms. Finally, we demonstrate that incorporating CICs into the domain invariant projection (DIP) algorithm can address its failure scenario caused by label-flipping features. We support our new algorithms and theoretical findings via numerical experiments on synthetic data, MNIST, CelebA, Camelyon17, and DomainNet datasets.