The Singular Values of Convolutional Layers

TL;DR

This paper characterizes the singular values of 2D convolutional layers, providing efficient computation methods and demonstrating that operator-norm regularization improves neural network test accuracy.

Contribution

It introduces a novel characterization of convolutional layer singular values and an algorithm for operator-norm projection, enhancing regularization techniques.

Findings

Operator-norm regularization improves test accuracy on CIFAR-10.

Efficient computation of singular values for convolutional layers.

Regularization reduces test error from 6.2% to 5.3%.

Abstract

We characterize the singular values of the linear transformation associated with a standard 2D multi-channel convolutional layer, enabling their efficient computation. This characterization also leads to an algorithm for projecting a convolutional layer onto an operator-norm ball. We show that this is an effective regularizer; for example, it improves the test error of a deep residual network using batch normalization on CIFAR-10 from 6.2\% to 5.3\%.