On regularization for a convolutional kernel in neural networks

TL;DR

This paper introduces a penalty function for convolutional kernels in neural networks that constrains their singular values around one, aiming to improve stability and generalization.

Contribution

It proposes a novel penalty function and an algorithm for optimizing convolutional kernels to maintain their singular values near one.

Findings

The method effectively constrains singular values in practice.

Numerical examples show improved stability.

Enhances generalization of neural networks.

Abstract

Convolutional neural network is an important model in deep learning. To avoid exploding/vanishing gradient problems and to improve the generalizability of a neural network, it is desirable to have a convolution operation that nearly preserves the norm, or to have the singular values of the transformation matrix corresponding to a convolutional kernel bounded around $1$. We propose a penalty function that can be used in the optimization of a convolutional neural network to constrain the singular values of the transformation matrix around $1$. We derive an algorithm to carry out the gradient descent minimization of this penalty function in terms of convolution kernels. Numerical examples are presented to demonstrate the effectiveness of the method.