Backward Gradient Normalization in Deep Neural Networks

TL;DR

This paper presents a novel gradient normalization method for deep neural networks that uses special layers during backpropagation to prevent vanishing or exploding gradients, leading to improved training and accuracy.

Contribution

It introduces backward gradient normalization layers that control gradient flow without affecting forward pass, enabling training of very deep networks.

Findings

Effective control of gradient norms in deep networks

Improved accuracy on several experimental conditions

Enables training of deeper neural architectures

Abstract

We introduce a new technique for gradient normalization during neural network training. The gradients are rescaled during the backward pass using normalization layers introduced at certain points within the network architecture. These normalization nodes do not affect forward activity propagation, but modify backpropagation equations to permit a well-scaled gradient flow that reaches the deepest network layers without experimenting vanishing or explosion. Results on tests with very deep neural networks show that the new technique can do an effective control of the gradient norm, allowing the update of weights in the deepest layers and improving network accuracy on several experimental conditions.