A Gentle Tutorial of Recurrent Neural Network with Error Backpropagation

TL;DR

This paper provides a clear tutorial on recurrent neural networks, focusing on the error backpropagation process, especially for LSTM units, to help understand their training mechanisms.

Contribution

It offers a detailed, accessible explanation of backpropagation in RNNs and LSTMs, clarifying complex concepts for learners and practitioners.

Findings

Clarifies the backpropagation process in RNNs and LSTMs

Provides step-by-step guidance on unfolding LSTM units

Enhances understanding of training recurrent neural networks

Abstract

We describe recurrent neural networks (RNNs), which have attracted great attention on sequential tasks, such as handwriting recognition, speech recognition and image to text. However, compared to general feedforward neural networks, RNNs have feedback loops, which makes it a little hard to understand the backpropagation step. Thus, we focus on basics, especially the error backpropagation to compute gradients with respect to model parameters. Further, we go into detail on how error backpropagation algorithm is applied on long short-term memory (LSTM) by unfolding the memory unit.