On the Hardness of Robustness Transfer: A Perspective from Rademacher Complexity over Symmetric Difference Hypothesis Space

TL;DR

This paper investigates the fundamental difficulty of adversarially robust domain adaptation by analyzing the adversarial Rademacher complexity over the symmetric difference hypothesis space, revealing intrinsic hardness and potential positive relations with standard domain adaptation.

Contribution

It introduces the analysis of adversarial Rademacher complexity over the symmetric difference hypothesis space, providing upper bounds and extending results to neural networks, thus advancing the theoretical understanding of robust domain adaptation.

Findings

Adversarial Rademacher complexity is always greater than the non-adversarial one for linear models.

Upper bounds on adversarial Rademacher complexity are established for linear and neural network models.

A positive relation between robust learning and standard domain adaptation is identified.

Abstract

Recent studies demonstrated that the adversarially robust learning under $\ell_\infty$ attack is harder to generalize to different domains than standard domain adaptation. How to transfer robustness across different domains has been a key question in domain adaptation field. To investigate the fundamental difficulty behind adversarially robust domain adaptation (or robustness transfer), we propose to analyze a key complexity measure that controls the cross-domain generalization: the adversarial Rademacher complexity over {\em symmetric difference hypothesis space} $\mathcal{H} \Delta \mathcal{H}$. For linear models, we show that adversarial version of this complexity is always greater than the non-adversarial one, which reveals the intrinsic hardness of adversarially robust domain adaptation. We also establish upper bounds on this complexity measure. Then we extend them to the ReLU neural network class by upper bounding the adversarial Rademacher complexity in the binary classification setting. Finally, even though the robust domain adaptation is provably harder, we do find positive relation between robust learning and standard domain adaptation. We explain \emph{how adversarial training helps domain adaptation in terms of standard risk}. We believe our results initiate the study of the generalization theory of adversarially robust domain adaptation, and could shed lights on distributed adversarially robust learning from heterogeneous sources, e.g., federated learning scenario.