Gating Revisited: Deep Multi-layer RNNs That Can Be Trained

TL;DR

This paper introduces the STAR cell, a new recurrent unit that is more parameter-efficient and robust for training deep RNNs, leading to better performance on sequence tasks.

Contribution

The paper presents the STAR cell, a novel gated recurrent unit that enables training deeper RNNs with fewer parameters and improved gradient stability.

Findings

STAR outperforms LSTM and GRU in deep RNN architectures.

Deep RNNs with STAR achieve higher accuracy on sequence modeling tasks.

STAR reduces computational resources needed for training deep RNNs.

Abstract

We propose a new STAckable Recurrent cell (STAR) for recurrent neural networks (RNNs), which has fewer parameters than widely used LSTM and GRU while being more robust against vanishing or exploding gradients. Stacking recurrent units into deep architectures suffers from two major limitations: (i) many recurrent cells (e.g., LSTMs) are costly in terms of parameters and computation resources; and (ii) deep RNNs are prone to vanishing or exploding gradients during training. We investigate the training of multi-layer RNNs and examine the magnitude of the gradients as they propagate through the network in the "vertical" direction. We show that, depending on the structure of the basic recurrent unit, the gradients are systematically attenuated or amplified. Based on our analysis we design a new type of gated cell that better preserves gradient magnitude. We validate our design on a large number of sequence modelling tasks and demonstrate that the proposed STAR cell allows to build and train deeper recurrent architectures, ultimately leading to improved performance while being computationally more efficient.