Asymptotic Singular Value Distribution of Linear Convolutional Layers

TL;DR

This paper analyzes the asymptotic spectral properties of linear convolutional layers in neural networks, proposing a more accurate singular value approximation method and spectral norm bounds that enhance regularization and generalization.

Contribution

It introduces a spectral density-based approximation for singular values of convolutional layers, improving accuracy over circular approximations and providing effective spectral norm bounds for regularization.

Findings

Spectral density approximation improves singular value estimation accuracy.

Spectral norm bounds serve as effective regularizers for ResNets.

Moderate improvement over circular approximation with subtle adjustments.

Abstract

In convolutional neural networks, the linear transformation of multi-channel two-dimensional convolutional layers with linear convolution is a block matrix with doubly Toeplitz blocks. Although a "wrapping around" operation can transform linear convolution to a circular one, by which the singular values can be approximated with reduced computational complexity by those of a block matrix with doubly circulant blocks, the accuracy of such an approximation is not guaranteed. In this paper, we propose to inspect such a linear transformation matrix through its asymptotic spectral representation - the spectral density matrix - by which we develop a simple singular value approximation method with improved accuracy over the circular approximation, as well as upper bounds for spectral norm with reduced computational complexity. Compared with the circular approximation, we obtain moderate improvement with a subtle adjustment of the singular value distribution. We also demonstrate that the spectral norm upper bounds are effective spectral regularizers for improving generalization performance in ResNets.