Fast Saturating Gate for Learning Long Time Scales with Recurrent Neural Networks

TL;DR

This paper introduces a novel fast gate function for recurrent neural networks that accelerates convergence, mitigates gradient vanishing, and improves learning of extremely long time scales in time series data.

Contribution

The paper proposes a new fast gate function with doubly exponential convergence, enhancing training efficiency and accuracy for long time scale modeling in RNNs.

Findings

Outperforms previous methods in accuracy on long time scale tasks.

Achieves better computational efficiency.

Effectively mitigates gradient vanishing in gate functions.

Abstract

Gate functions in recurrent models, such as an LSTM and GRU, play a central role in learning various time scales in modeling time series data by using a bounded activation function. However, it is difficult to train gates to capture extremely long time scales due to gradient vanishing of the bounded function for large inputs, which is known as the saturation problem. We closely analyze the relation between saturation of the gate function and efficiency of the training. We prove that the gradient vanishing of the gate function can be mitigated by accelerating the convergence of the saturating function, i.e., making the output of the function converge to 0 or 1 faster. Based on the analysis results, we propose a gate function called fast gate that has a doubly exponential convergence rate with respect to inputs by simple function composition. We empirically show that our method outperforms previous methods in accuracy and computational efficiency on benchmark tasks involving extremely long time scales.