Wasserstein distributional robustness of neural networks

TL;DR

This paper introduces a Wasserstein distributionally robust optimization framework for neural networks, proposing new attack algorithms and providing bounds on adversarial accuracy under distributional threats, with empirical validation on CIFAR-10.

Contribution

It develops a novel Wasserstein DRO approach for neural network robustness, including new attack algorithms and asymptotic accuracy bounds, extending beyond traditional pointwise attacks.

Findings

Proposed first-order attack algorithms including FGSM and PGD as special cases.

Provided a fast, first-order accurate asymptotic estimate of adversarial accuracy.

Validated theoretical results with experiments on CIFAR-10 dataset.

Abstract

Deep neural networks are known to be vulnerable to adversarial attacks (AA). For an image recognition task, this means that a small perturbation of the original can result in the image being misclassified. Design of such attacks as well as methods of adversarial training against them are subject of intense research. We re-cast the problem using techniques of Wasserstein distributionally robust optimization (DRO) and obtain novel contributions leveraging recent insights from DRO sensitivity analysis. We consider a set of distributional threat models. Unlike the traditional pointwise attacks, which assume a uniform bound on perturbation of each input data point, distributional threat models allow attackers to perturb inputs in a non-uniform way. We link these more general attacks with questions of out-of-sample performance and Knightian uncertainty. To evaluate the distributional robustness of neural networks, we propose a first-order AA algorithm and its multi-step version. Our attack algorithms include Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD) as special cases. Furthermore, we provide a new asymptotic estimate of the adversarial accuracy against distributional threat models. The bound is fast to compute and first-order accurate, offering new insights even for the pointwise AA. It also naturally yields out-of-sample performance guarantees. We conduct numerical experiments on the CIFAR-10 dataset using DNNs on RobustBench to illustrate our theoretical results. Our code is available at https://github.com/JanObloj/W-DRO-Adversarial-Methods.