Provable Robustness of Adversarial Training for Learning Halfspaces with Noise

TL;DR

This paper provides theoretical guarantees for adversarial training methods in learning robust halfspaces under label noise, showing they achieve provable robustness bounds for different perturbation norms.

Contribution

It offers the first provable analysis of adversarial training for robust halfspaces with noise, establishing bounds for both convex and nonconvex loss functions.

Findings

Adversarial training achieves robust error bounds proportional to the optimal robust error.

Results apply to distributions with anti-concentration properties like log-concave distributions.

Nonconvex sigmoidal loss improves robustness bounds compared to standard convex loss.

Abstract

We analyze the properties of adversarial training for learning adversarially robust halfspaces in the presence of agnostic label noise. Denoting $\mathsf{OPT}_{p,r}$ as the best robust classification error achieved by a halfspace that is robust to perturbations of $\ell_{p}$ balls of radius $r$, we show that adversarial training on the standard binary cross-entropy loss yields adversarially robust halfspaces up to (robust) classification error $\tilde O(\sqrt{\mathsf{OPT}_{2,r}})$ for $p=2$, and $\tilde O(d^{1/4} \sqrt{\mathsf{OPT}_{\infty, r}} + d^{1/2} \mathsf{OPT}_{\infty,r})$ when $p=\infty$. Our results hold for distributions satisfying anti-concentration properties enjoyed by log-concave isotropic distributions among others. We additionally show that if one instead uses a nonconvex sigmoidal loss, adversarial training yields halfspaces with an improved robust classification error of $O(\mathsf{OPT}_{2,r})$ for $p=2$, and $O(d^{1/4}\mathsf{OPT}_{\infty, r})$ when $p=\infty$. To the best of our knowledge, this is the first work to show that adversarial training provably yields robust classifiers in the presence of noise.