Curse of Small Sample Size in Forecasting of the Active Cases in COVID-19 Outbreak

TL;DR

This study investigates why machine learning models struggle to accurately forecast COVID-19 active cases long-term, highlighting the impact of small sample sizes and demonstrating that simple models perform best within a short 3-day window.

Contribution

The paper analyzes the limitations of complex machine learning models in COVID-19 forecasting due to small sample sizes and compares feature selection methods and models to improve short-term prediction accuracy.

Findings

Linear regression achieves high accuracy for 2-week forecasts

Complex models fail to generalize well with limited data

Short-term predictions (up to 3 days) are feasible with current data

Abstract

During the COVID-19 pandemic, a massive number of attempts on the predictions of the number of cases and the other future trends of this pandemic have been made. However, they fail to predict, in a reliable way, the medium and long term evolution of fundamental features of COVID-19 outbreak within acceptable accuracy. This paper gives an explanation for the failure of machine learning models in this particular forecasting problem. The paper shows that simple linear regression models provide high prediction accuracy values reliably but only for a 2-weeks period and that relatively complex machine learning models, which have the potential of learning long term predictions with low errors, cannot achieve to obtain good predictions with possessing a high generalization ability. It is suggested in the paper that the lack of a sufficient number of samples is the source of low prediction performance of the forecasting models. The reliability of the forecasting results about the active cases is measured in terms of the cross-validation prediction errors, which are used as expectations for the generalization errors of the forecasters. To exploit the information, which is of most relevant with the active cases, we perform feature selection over a variety of variables. We apply different feature selection methods, namely the Pairwise Correlation, Recursive Feature Selection, and feature selection by using the Lasso regression and compare them to each other and also with the models not employing any feature selection. Furthermore, we compare Linear Regression, Multi-Layer Perceptron, and Long-Short Term Memory models each of which is used for prediction active cases together with the mentioned feature selection methods. Our results show that the accurate forecasting of the active cases with high generalization ability is possible up to 3 days only because of the small sample size of COVID-19 data.