Simple Recurrent Units for Highly Parallelizable Recurrence

TL;DR

The paper introduces the Simple Recurrent Unit (SRU), a lightweight, highly parallelizable recurrent neural network component that improves training speed and performance on NLP tasks compared to traditional LSTM and Transformer models.

Contribution

The paper presents the SRU, a novel recurrent unit that balances expressiveness and scalability, enabling faster training and better results than existing models.

Findings

SRU achieves 5-9x speed-up over cuDNN LSTM.

SRU outperforms LSTM and convolutional models on NLP tasks.

Incorporating SRU into Transformer improves BLEU score by 0.7 on translation.

Abstract

Common recurrent neural architectures scale poorly due to the intrinsic difficulty in parallelizing their state computations. In this work, we propose the Simple Recurrent Unit (SRU), a light recurrent unit that balances model capacity and scalability. SRU is designed to provide expressive recurrence, enable highly parallelized implementation, and comes with careful initialization to facilitate training of deep models. We demonstrate the effectiveness of SRU on multiple NLP tasks. SRU achieves 5--9x speed-up over cuDNN-optimized LSTM on classification and question answering datasets, and delivers stronger results than LSTM and convolutional models. We also obtain an average of 0.7 BLEU improvement over the Transformer model on translation by incorporating SRU into the architecture.