A single gradient step finds adversarial examples on random two-layers neural networks

TL;DR

This paper demonstrates that a single gradient step can find adversarial examples in overcomplete two-layer neural networks with smooth activations, extending previous results from undercomplete networks.

Contribution

It extends the understanding of adversarial example generation to overcomplete neural networks, showing a single gradient step suffices in this broader setting.

Findings

Single gradient step finds adversarial examples in overcomplete networks

Results hold for subexponential width networks with smooth activations

Extends previous undercomplete network results to overcomplete case

Abstract

Daniely and Schacham recently showed that gradient descent finds adversarial examples on random undercomplete two-layers ReLU neural networks. The term "undercomplete" refers to the fact that their proof only holds when the number of neurons is a vanishing fraction of the ambient dimension. We extend their result to the overcomplete case, where the number of neurons is larger than the dimension (yet also subexponential in the dimension). In fact we prove that a single step of gradient descent suffices. We also show this result for any subexponential width random neural network with smooth activation function.