Statistical Analysis of Conditional Group Distributionally Robust Optimization with Cross-Entropy Loss

TL;DR

This paper introduces a novel Conditional Group Distributionally Robust Optimization framework for multi-source domain adaptation, providing theoretical guarantees and a perturbation-based inference method to handle distribution shifts and nonstandard asymptotics.

Contribution

The paper develops a new CG-DRO method with efficient algorithms, establishes convergence rates, and proposes a perturbation-based inference procedure for reliable uncertainty quantification.

Findings

Fast statistical convergence rates for CG-DRO estimator.

Effective perturbation-based inference for nonstandard asymptotics.

Theoretical guarantees for distribution shift robustness.

Abstract

In multi-source learning with discrete labels, distributional heterogeneity across domains poses a central challenge to developing predictive models that transfer reliably to unseen domains. We study multi-source unsupervised domain adaptation, where labeled data are available from multiple source domains and only unlabeled data are observed from the target domain. To address potential distribution shifts, we propose a novel Conditional Group Distributionally Robust Optimization (CG-DRO) framework that learns a classifier by minimizing the worst-case cross-entropy loss over the convex combinations of the conditional outcome distributions from sources domains. We develop an efficient Mirror Prox algorithm for solving the minimax problem and employ a double machine learning procedure to estimate the risk function, ensuring that errors in nuisance estimation contribute only at higher-order rates. We establish fast statistical convergence rates for the empirical CG-DRO estimator by constructing two surrogate minimax optimization problems that serve as theoretical bridges. A distinguishing challenge for CG-DRO is the emergence of nonstandard asymptotics: the empirical CG-DRO estimator may fail to converge to a standard limiting distribution due to boundary effects and system instability. To address this, we introduce a perturbation-based inference procedure that enables uniformly valid inference, including confidence interval construction and hypothesis testing.