Scaling provable adversarial defenses

TL;DR

This paper advances scalable provable adversarial defenses for larger neural networks by extending training methods to complex architectures, introducing efficient nonlinear random projections, and employing cascade models, achieving state-of-the-art robustness on MNIST and CIFAR.

Contribution

It introduces a modular approach for training robust networks with skip connections, a linear-scaling nonlinear random projection method for ReLU networks, and a cascade model technique to improve robustness.

Findings

Achieved 3.1% robust error on MNIST with $\, ext{l}_ ext{infty}$ perturbations.

Reduced robust error on CIFAR from 80% to 36.4%.

Extended provable defenses to larger, more complex network architectures.

Abstract

Recent work has developed methods for learning deep network classifiers that are provably robust to norm-bounded adversarial perturbation; however, these methods are currently only possible for relatively small feedforward networks. In this paper, in an effort to scale these approaches to substantially larger models, we extend previous work in three main directions. First, we present a technique for extending these training procedures to much more general networks, with skip connections (such as ResNets) and general nonlinearities; the approach is fully modular, and can be implemented automatically (analogous to automatic differentiation). Second, in the specific case of $\ell_\infty$ adversarial perturbations and networks with ReLU nonlinearities, we adopt a nonlinear random projection for training, which scales linearly in the number of hidden units (previous approaches scaled quadratically). Third, we show how to further improve robust error through cascade models. On both MNIST and CIFAR data sets, we train classifiers that improve substantially on the state of the art in provable robust adversarial error bounds: from 5.8% to 3.1% on MNIST (with $\ell_\infty$ perturbations of $\epsilon=0.1$), and from 80% to 36.4% on CIFAR (with $\ell_\infty$ perturbations of $\epsilon=2/255$). Code for all experiments in the paper is available at https://github.com/locuslab/convex_adversarial/.