A Critical Review of Recurrent Neural Networks for Sequence Learning

TL;DR

This paper reviews the development and advancements of recurrent neural networks (RNNs), especially LSTM and bidirectional architectures, highlighting their success in sequence learning tasks across various domains over the past three decades.

Contribution

It provides a comprehensive synthesis of RNN research history, clarifies conflicting notation, and explains recent breakthroughs making RNNs practically viable for large-scale sequence tasks.

Findings

RNNs effectively model sequential data across multiple domains.

Advances in architectures and training techniques have improved RNN performance.

LSTM and bidirectional RNNs have achieved state-of-the-art results in tasks like translation and recognition.

Abstract

Countless learning tasks require dealing with sequential data. Image captioning, speech synthesis, and music generation all require that a model produce outputs that are sequences. In other domains, such as time series prediction, video analysis, and musical information retrieval, a model must learn from inputs that are sequences. Interactive tasks, such as translating natural language, engaging in dialogue, and controlling a robot, often demand both capabilities. Recurrent neural networks (RNNs) are connectionist models that capture the dynamics of sequences via cycles in the network of nodes. Unlike standard feedforward neural networks, recurrent networks retain a state that can represent information from an arbitrarily long context window. Although recurrent neural networks have traditionally been difficult to train, and often contain millions of parameters, recent advances in network architectures, optimization techniques, and parallel computation have enabled successful large-scale learning with them. In recent years, systems based on long short-term memory (LSTM) and bidirectional (BRNN) architectures have demonstrated ground-breaking performance on tasks as varied as image captioning, language translation, and handwriting recognition. In this survey, we review and synthesize the research that over the past three decades first yielded and then made practical these powerful learning models. When appropriate, we reconcile conflicting notation and nomenclature. Our goal is to provide a self-contained explication of the state of the art together with a historical perspective and references to primary research.