A Simple Way to Initialize Recurrent Networks of Rectified Linear Units

TL;DR

This paper introduces a straightforward initialization method for recurrent neural networks with ReLU units, using identity matrices, which achieves performance comparable to LSTM on various benchmarks.

Contribution

The paper proposes a simple initialization technique for ReLU-based RNNs that enhances learning of long-term dependencies without complex architectures.

Findings

Comparable performance to LSTM on four benchmarks

Effective in long-range temporal tasks

Simplifies recurrent network training

Abstract

Learning long term dependencies in recurrent networks is difficult due to vanishing and exploding gradients. To overcome this difficulty, researchers have developed sophisticated optimization techniques and network architectures. In this paper, we propose a simpler solution that use recurrent neural networks composed of rectified linear units. Key to our solution is the use of the identity matrix or its scaled version to initialize the recurrent weight matrix. We find that our solution is comparable to LSTM on our four benchmarks: two toy problems involving long-range temporal structures, a large language modeling problem and a benchmark speech recognition problem.