Regularization for convolutional kernel tensors to avoid unstable gradient problem in convolutional neural networks

TL;DR

This paper introduces three regularization techniques for convolutional kernels in neural networks to control singular values, aiming to prevent unstable gradients during training.

Contribution

It proposes novel regularization terms for convolutional kernels that help stabilize training by constraining singular values of transformation matrices.

Findings

Regularization improves training stability.

Singular value constraints reduce gradient explosion/vanishing.

New gradient methods facilitate implementation.

Abstract

Convolutional neural networks are very popular nowadays. Training neural networks is not an easy task. Each convolution corresponds to a structured transformation matrix. In order to help avoid the exploding/vanishing gradient problem, it is desirable that the singular values of each transformation matrix are not large/small in the training process. We propose three new regularization terms for a convolutional kernel tensor to constrain the singular values of each transformation matrix. We show how to carry out the gradient type methods, which provides new insight about the training of convolutional neural networks.