Machine learning in an expectation-maximisation framework for nowcasting

TL;DR

This paper introduces a machine learning-enhanced expectation-maximisation framework for nowcasting, effectively modeling incomplete information like reporting delays using neural networks and gradient boosting, demonstrated through simulations and COVID-19 data.

Contribution

It develops a novel EM-based nowcasting method incorporating machine learning models for better handling of high-dimensional, non-linear data in reporting delays.

Findings

Machine learning models outperform traditional methods in simulation studies.

XGBoost-based approach is most effective in real COVID-19 data.

Framework successfully models occurrence and reporting processes with covariates.

Abstract

Decision making often occurs in the presence of incomplete information, leading to the under- or overestimation of risk. Leveraging the observable information to learn the complete information is called nowcasting. In practice, incomplete information is often a consequence of reporting or observation delays. In this paper, we propose an expectation-maximisation (EM) framework for nowcasting that uses machine learning techniques to model both the occurrence as well as the reporting process of events. We allow for the inclusion of covariate information specific to the occurrence and reporting periods as well as characteristics related to the entity for which events occurred. We demonstrate how the maximisation step and the information flow between EM iterations can be tailored to leverage the predictive power of neural networks and (extreme) gradient boosting machines (XGBoost). With simulation experiments, we show that we can effectively model both the occurrence and reporting of events when dealing with high-dimensional covariate information. In the presence of non-linear effects, we show that our methodology outperforms existing EM-based nowcasting frameworks that use generalised linear models in the maximisation step. Finally, we apply the framework to the reporting of Argentinian Covid-19 cases, where the XGBoost-based approach again is most performant.