Least Redundant Gated Recurrent Neural Network

TL;DR

This paper introduces the Deep Memory Update (DMU), a novel recurrent neural network architecture that enhances training stability and efficiency while competing with or outperforming existing models like LSTM and GRU.

Contribution

The paper presents DMU, a new RNN architecture with deep state transformation, improved training stability, and competitive performance against state-of-the-art models.

Findings

DMU trains stably and efficiently.

DMU outperforms LSTM, GRU, and Recurrent Highway Networks in experiments.

DMU can learn complex state transformations.

Abstract

Recurrent neural networks are important tools for sequential data processing. However, they are notorious for problems regarding their training. Challenges include capturing complex relations between consecutive states and stability and efficiency of training. In this paper, we introduce a recurrent neural architecture called Deep Memory Update (DMU). It is based on updating the previous memory state with a deep transformation of the lagged state and the network input. The architecture is able to learn to transform its internal state using any nonlinear function. Its training is stable and fast due to relating its learning rate to the size of the module. Even though DMU is based on standard components, experimental results presented here confirm that it can compete with and often outperform state-of-the-art architectures such as Long Short-Term Memory, Gated Recurrent Units, and Recurrent Highway Networks.