Long Short-term Memory RNN

TL;DR

This paper introduces the architecture and foundational formulas of LSTM cells, discusses their applications in time-series forecasting and NLP, and compares them with traditional statistical methods like ARIMA.

Contribution

It provides a detailed explanation of LSTM components, their mathematical foundations, and practical insights into their strengths and weaknesses in various domains.

Findings

LSTMs effectively model complex time-series data.

LSTMs outperform traditional methods like ARIMA in certain tasks.

Strengths and weaknesses of LSTMs are discussed in practical contexts.

Abstract

This paper is based on a machine learning project at the Norwegian University of Science and Technology, fall 2020. The project was initiated with a literature review on the latest developments within time-series forecasting methods in the scientific community over the past five years. The paper summarizes the essential aspects of this research. Furthermore, in this paper, we introduce an LSTM cell's architecture, and explain how different components go together to alter the cell's memory and predict the output. Also, the paper provides the necessary formulas and foundations to calculate a forward iteration through an LSTM. Then, the paper refers to some practical applications and research that emphasize the strength and weaknesses of LSTMs, shown within the time-series domain and the natural language processing (NLP) domain. Finally, alternative statistical methods for time series predictions are highlighted, where the paper outline ARIMA and exponential smoothing. Nevertheless, as LSTMs can be viewed as a complex architecture, the paper assumes that the reader has some knowledge of essential machine learning aspects, such as the multi-layer perceptron, activation functions, overfitting, backpropagation, bias, over- and underfitting, and more.