Deep Independently Recurrent Neural Network (IndRNN)

TL;DR

This paper introduces the Independently Recurrent Neural Network (IndRNN), a new RNN variant that effectively learns long-term dependencies, is easier to train, and is significantly faster than LSTM, enabling very deep architectures and processing long sequences.

Contribution

The paper proposes IndRNN, a novel RNN architecture with independent neurons that addresses gradient issues and enables deeper, faster networks using non-saturated activations.

Findings

IndRNN effectively solves gradient vanishing and exploding problems.

IndRNN can be much deeper than traditional RNNs.

IndRNN outperforms RNN, LSTM, and Transformer on various tasks.

Abstract

Recurrent neural networks (RNNs) are known to be difficult to train due to the gradient vanishing and exploding problems and thus difficult to learn long-term patterns and construct deep networks. To address these problems, this paper proposes a new type of RNNs with the recurrent connection formulated as Hadamard product, referred to as independently recurrent neural network (IndRNN), where neurons in the same layer are independent of each other and connected across layers. Due to the better behaved gradient backpropagation, IndRNN with regulated recurrent weights effectively addresses the gradient vanishing and exploding problems and thus long-term dependencies can be learned. Moreover, an IndRNN can work with non-saturated activation functions such as ReLU (rectified linear unit) and be still trained robustly. Different deeper IndRNN architectures, including the basic stacked IndRNN, residual IndRNN and densely connected IndRNN, have been investigated, all of which can be much deeper than the existing RNNs. Furthermore, IndRNN reduces the computation at each time step and can be over 10 times faster than the commonly used Long short-term memory (LSTM). Experimental results have shown that the proposed IndRNN is able to process very long sequences and construct very deep networks. Better performance has been achieved on various tasks with IndRNNs compared with the traditional RNN, LSTM and the popular Transformer.