Improving Out-of-Distribution Generalization by Adversarial Training with Structured Priors

TL;DR

This paper introduces structured adversarial training methods with low-rank perturbations to improve out-of-distribution generalization in deep models, outperforming traditional approaches on benchmark tests.

Contribution

It proposes two novel adversarial training variants utilizing low-rank structures to enhance robustness against large distribution shifts in OOD scenarios.

Findings

Universal adversarial training is more robust to large perturbations.

Structured low-rank perturbations outperform sample-wise adversarial training.

Proposed methods achieve better OOD performance on DomainBed benchmark.

Abstract

Deep models often fail to generalize well in test domains when the data distribution differs from that in the training domain. Among numerous approaches to address this Out-of-Distribution (OOD) generalization problem, there has been a growing surge of interest in exploiting Adversarial Training (AT) to improve OOD performance. Recent works have revealed that the robust model obtained by conducting sample-wise AT also retains transferability to biased test domains. In this paper, we empirically show that sample-wise AT has limited improvement on OOD performance. Specifically, we find that AT can only maintain performance at smaller scales of perturbation while Universal AT (UAT) is more robust to larger-scale perturbations. This provides us with clues that adversarial perturbations with universal (low dimensional) structures can enhance the robustness against large data distribution shifts that are common in OOD scenarios. Inspired by this, we propose two AT variants with low-rank structures to train OOD-robust models. Extensive experiments on DomainBed benchmark show that our proposed approaches outperform Empirical Risk Minimization (ERM) and sample-wise AT. Our code is available at https://github.com/NOVAglow646/NIPS22-MAT-and-LDAT-for-OOD.