Pay attention to your loss: understanding misconceptions about 1-Lipschitz neural networks

TL;DR

This paper clarifies misconceptions about 1-Lipschitz neural networks, demonstrating they can be as accurate as classical networks, generalize well, and offer advantages in robustness and training control.

Contribution

The paper provides a comprehensive analysis showing 1-Lipschitz networks are as accurate as unconstrained ones, identify the most robust classifier, and highlight the importance of hyper-parameters.

Findings

1-Lipschitz networks match classical accuracy levels

They generalize well under mild assumptions

Hyper-parameters control accuracy-robustness trade-off

Abstract

Lipschitz constrained networks have gathered considerable attention in the deep learning community, with usages ranging from Wasserstein distance estimation to the training of certifiably robust classifiers. However they remain commonly considered as less accurate, and their properties in learning are still not fully understood. In this paper we clarify the matter: when it comes to classification 1-Lipschitz neural networks enjoy several advantages over their unconstrained counterpart. First, we show that these networks are as accurate as classical ones, and can fit arbitrarily difficult boundaries. Then, relying on a robustness metric that reflects operational needs we characterize the most robust classifier: the WGAN discriminator. Next, we show that 1-Lipschitz neural networks generalize well under milder assumptions. Finally, we show that hyper-parameters of the loss are crucial for controlling the accuracy-robustness trade-off. We conclude that they exhibit appealing properties to pave the way toward provably accurate, and provably robust neural networks.