L2-Nonexpansive Neural Networks

TL;DR

This paper introduces L2-nonexpansive neural networks that are inherently robust to adversarial attacks and noisy data, achieved through a novel regularization scheme and improved control of Lipschitz constants.

Contribution

It presents a new class of well-conditioned neural networks with enhanced robustness, including a regularization method, adapted nonlinearities, and a new loss function, without adversarial training.

Findings

Outperform state-of-the-art in L2 adversarial robustness on MNIST and CIFAR-10

Generalize better from noisy data with random labels

Outputs provide meaningful confidence and generalization measures

Abstract

This paper proposes a class of well-conditioned neural networks in which a unit amount of change in the inputs causes at most a unit amount of change in the outputs or any of the internal layers. We develop the known methodology of controlling Lipschitz constants to realize its full potential in maximizing robustness, with a new regularization scheme for linear layers, new ways to adapt nonlinearities and a new loss function. With MNIST and CIFAR-10 classifiers, we demonstrate a number of advantages. Without needing any adversarial training, the proposed classifiers exceed the state of the art in robustness against white-box L2-bounded adversarial attacks. They generalize better than ordinary networks from noisy data with partially random labels. Their outputs are quantitatively meaningful and indicate levels of confidence and generalization, among other desirable properties.