RLSbench: Domain Adaptation Under Relaxed Label Shift

TL;DR

This paper introduces RLSbench, a comprehensive benchmark for relaxed label shift across multiple data modalities, revealing limitations of current methods and proposing a meta-algorithm that significantly improves adaptation performance under label proportion shifts.

Contribution

The paper presents RLSbench, a large-scale benchmark for relaxed label shift, and develops a meta-algorithm that enhances domain adaptation methods under label proportion shifts.

Findings

Many existing domain adaptation methods fail under label proportion shifts.

The proposed meta-algorithm improves accuracy by 2-10% in shifted settings.

RLSbench enables rigorous evaluation of domain adaptation methods in relaxed label shift scenarios.

Abstract

Despite the emergence of principled methods for domain adaptation under label shift, their sensitivity to shifts in class conditional distributions is precariously under explored. Meanwhile, popular deep domain adaptation heuristics tend to falter when faced with label proportions shifts. While several papers modify these heuristics in attempts to handle label proportions shifts, inconsistencies in evaluation standards, datasets, and baselines make it difficult to gauge the current best practices. In this paper, we introduce RLSbench, a large-scale benchmark for relaxed label shift, consisting of $>$500 distribution shift pairs spanning vision, tabular, and language modalities, with varying label proportions. Unlike existing benchmarks, which primarily focus on shifts in class-conditional $p(x|y)$, our benchmark also focuses on label marginal shifts. First, we assess 13 popular domain adaptation methods, demonstrating more widespread failures under label proportion shifts than were previously known. Next, we develop an effective two-step meta-algorithm that is compatible with most domain adaptation heuristics: (i) pseudo-balance the data at each epoch; and (ii) adjust the final classifier with target label distribution estimate. The meta-algorithm improves existing domain adaptation heuristics under large label proportion shifts, often by 2--10\% accuracy points, while conferring minimal effect ($<$0.5\%) when label proportions do not shift. We hope that these findings and the availability of RLSbench will encourage researchers to rigorously evaluate proposed methods in relaxed label shift settings. Code is publicly available at https://github.com/acmi-lab/RLSbench.