Tight and Efficient Upper Bound on Spectral Norm of Convolutional Layers

TL;DR

This paper introduces a new, efficient upper bound for the spectral norm of convolutional layers that is independent of input size, aiding in training stability and performance of CNNs.

Contribution

The authors derive a tensor-based spectral norm upper bound for convolution kernels that is both tight and computationally efficient, improving over existing methods.

Findings

The new bound is independent of input image resolution.

It can be computed efficiently during training.

Using the bound improves CNN performance.

Abstract

Controlling the spectral norm of the Jacobian matrix, which is related to the convolution operation, has been shown to improve generalization, training stability and robustness in CNNs. Existing methods for computing the norm either tend to overestimate it or their performance may deteriorate quickly with increasing the input and kernel sizes. In this paper, we demonstrate that the tensor version of the spectral norm of a four-dimensional convolution kernel, up to a constant factor, serves as an upper bound for the spectral norm of the Jacobian matrix associated with the convolution operation. This new upper bound is independent of the input image resolution, differentiable and can be efficiently calculated during training. Through experiments, we demonstrate how this new bound can be used to improve the performance of convolutional architectures.