Lipschitz Bound Analysis of Neural Networks

TL;DR

This paper investigates the challenges in estimating tight Lipschitz bounds for CNNs, highlighting a significant gap between theoretical bounds and actual network behavior through extensive empirical analysis.

Contribution

It identifies the difficulty in obtaining non-trivial Lipschitz bounds for CNNs, proposes methods to convert CNNs to fully connected networks, and quantifies the gap between actual and estimated bounds.

Findings

Significant gap (20x-50x) between actual and estimated Lipschitz constants.

Conversion of CNNs to fully connected networks via unrolling layers or Toeplitz matrices.

Extensive experiments on MNIST and CIFAR-10 datasets support the analysis.

Abstract

Lipschitz Bound Estimation is an effective method of regularizing deep neural networks to make them robust against adversarial attacks. This is useful in a variety of applications ranging from reinforcement learning to autonomous systems. In this paper, we highlight the significant gap in obtaining a non-trivial Lipschitz bound certificate for Convolutional Neural Networks (CNNs) and empirically support it with extensive graphical analysis. We also show that unrolling Convolutional layers or Toeplitz matrices can be employed to convert Convolutional Neural Networks (CNNs) to a Fully Connected Network. Further, we propose a simple algorithm to show the existing 20x-50x gap in a particular data distribution between the actual lipschitz constant and the obtained tight bound. We also ran sets of thorough experiments on various network architectures and benchmark them on datasets like MNIST and CIFAR-10. All these proposals are supported by extensive testing, graphs, histograms and comparative analysis.