Sampling-based Gradient Regularization for Capturing Long-Term Dependencies in Recurrent Neural Networks

TL;DR

This paper introduces a universal sampling-based gradient regularization technique for RNNs that maintains gradient norms within suitable ranges, enhancing the network's ability to learn long-term dependencies in sequences.

Contribution

The authors propose a novel method to estimate and control the contribution of each training example to the gradient norm, improving RNN training for long-term dependencies.

Findings

RNNs can detect event links over approximately 100 time steps.

The method maintains stable gradient norms during training.

Experimental results show improved long-term dependency learning.

Abstract

Vanishing (and exploding) gradients effect is a common problem for recurrent neural networks with nonlinear activation functions which use backpropagation method for calculation of derivatives. Deep feedforward neural networks with many hidden layers also suffer from this effect. In this paper we propose a novel universal technique that makes the norm of the gradient stay in the suitable range. We construct a way to estimate a contribution of each training example to the norm of the long-term components of the target function s gradient. Using this subroutine we can construct mini-batches for the stochastic gradient descent (SGD) training that leads to high performance and accuracy of the trained network even for very complex tasks. We provide a straightforward mathematical estimation of minibatch s impact on for the gradient norm and prove its correctness theoretically. To check our framework experimentally we use some special synthetic benchmarks for testing RNNs on ability to capture long-term dependencies. Our network can detect links between events in the (temporal) sequence at the range approx. 100 and longer.