Certifying Some Distributional Robustness with Principled Adversarial Training

TL;DR

This paper introduces a principled adversarial training method based on distributionally robust optimization within a Wasserstein ball, providing certified robustness guarantees with minimal additional computational cost.

Contribution

It proposes a novel training procedure that incorporates worst-case distributional perturbations, offering theoretical robustness guarantees and improved performance over heuristic methods.

Findings

Achieves moderate robustness with little extra cost

Certifies robustness for the population loss efficiently

Outperforms heuristic approaches for imperceptible perturbations

Abstract

Neural networks are vulnerable to adversarial examples and researchers have proposed many heuristic attack and defense mechanisms. We address this problem through the principled lens of distributionally robust optimization, which guarantees performance under adversarial input perturbations. By considering a Lagrangian penalty formulation of perturbing the underlying data distribution in a Wasserstein ball, we provide a training procedure that augments model parameter updates with worst-case perturbations of training data. For smooth losses, our procedure provably achieves moderate levels of robustness with little computational or statistical cost relative to empirical risk minimization. Furthermore, our statistical guarantees allow us to efficiently certify robustness for the population loss. For imperceptible perturbations, our method matches or outperforms heuristic approaches.