Lipschitz regularity of deep neural networks: analysis and efficient estimation

TL;DR

This paper introduces AutoLip and SeqLip, novel algorithms for efficiently estimating the Lipschitz constant of deep neural networks, which is crucial for assessing and improving their robustness and regularity.

Contribution

The paper presents the first generic algorithm AutoLip for upper bounding Lipschitz constants of differentiable functions and an improved method SeqLip for sequential networks, enhancing estimation accuracy.

Findings

AutoLip efficiently computes Lipschitz bounds using automatic differentiation.

SeqLip provides tighter bounds by exploiting network structure.

Experiments show SeqLip significantly improves existing upper bounds.

Abstract

Deep neural networks are notorious for being sensitive to small well-chosen perturbations, and estimating the regularity of such architectures is of utmost importance for safe and robust practical applications. In this paper, we investigate one of the key characteristics to assess the regularity of such methods: the Lipschitz constant of deep learning architectures. First, we show that, even for two layer neural networks, the exact computation of this quantity is NP-hard and state-of-art methods may significantly overestimate it. Then, we both extend and improve previous estimation methods by providing AutoLip, the first generic algorithm for upper bounding the Lipschitz constant of any automatically differentiable function. We provide a power method algorithm working with automatic differentiation, allowing efficient computations even on large convolutions. Second, for sequential neural networks, we propose an improved algorithm named SeqLip that takes advantage of the linear computation graph to split the computation per pair of consecutive layers. Third we propose heuristics on SeqLip in order to tackle very large networks. Our experiments show that SeqLip can significantly improve on the existing upper bounds. Finally, we provide an implementation of AutoLip in the PyTorch environment that may be used to better estimate the robustness of a given neural network to small perturbations or regularize it using more precise Lipschitz estimations.