Deep Transformer Model with Pre-Layer Normalization for COVID-19 Growth Prediction

TL;DR

This paper introduces a Deep Transformer model with pre-layer normalization for predicting COVID-19 case growth, demonstrating superior accuracy over LSTM and RNN models using MAPE as the evaluation metric.

Contribution

The study proposes a novel Deep Transformer architecture with pre-layer normalization for COVID-19 growth prediction, optimizing hyperparameters and outperforming traditional RNN-based models.

Findings

Deep Transformer with pre-layer normalization achieves MAPE of 18.83.

The Adamax optimizer yields the best performance.

Deep Transformer outperforms LSTM and RNN models.

Abstract

Coronavirus disease or COVID-19 is an infectious disease caused by the SARS-CoV-2 virus. The first confirmed case caused by this virus was found at the end of December 2019 in Wuhan City, China. This case then spread throughout the world, including Indonesia. Therefore, the COVID-19 case was designated as a global pandemic by WHO. The growth of COVID-19 cases, especially in Indonesia, can be predicted using several approaches, such as the Deep Neural Network (DNN). One of the DNN models that can be used is Deep Transformer which can predict time series. The model is trained with several test scenarios to get the best model. The evaluation is finding the best hyperparameters. Then, further evaluation was carried out using the best hyperparameters setting of the number of prediction days, the optimizer, the number of features, and comparison with the former models of the Long Short-Term Memory (LSTM) and Recurrent Neural Network (RNN). All evaluations used metric of the Mean Absolute Percentage Error (MAPE). Based on the results of the evaluations, Deep Transformer produces the best results when using the Pre-Layer Normalization and predicting one day ahead with a MAPE value of 18.83. Furthermore, the model trained with the Adamax optimizer obtains the best performance among other tested optimizers. The performance of the Deep Transformer also exceeds other test models, which are LSTM and RNN.