Performance evaluation of an operational dengue forecasting system (D-MOSS) in Vietnam

TL;DR

This paper evaluates a dengue forecasting system in Vietnam, showing it performs well even at longer forecast lead times.

Contribution

The study provides a comprehensive operational evaluation of a dengue forecasting system under real-world conditions.

Findings

D-MOSS outperformed baseline models across most performance metrics.

Forecast accuracy remained relatively high even at four to six month lead times.

Operational utility scenarios showed high accuracy in predicting outbreak thresholds.

Abstract

D-MOSS (Dengue forecasting Model Satellite-based System) was launched operationally in Vietnam in June 2019, providing near-real time dengue forecasts across all 63 provinces. Very few dengue forecasting systems have prospectively evaluated the performance of dengue forecasting under real-world operational conditions. This study comprehensively assesses D-MOSS dengue forecasting performance since operationalisation through both statistical accuracy (absolute dengue incidence, trajectory of incidence, timing of peaks), and operational utility (predictions for specific decision-making scenarios). The D-MOSS dengue forecasts in Vietnam outperformed null model baselines across almost all performance metrics.. While lead times of one month reported the highest accuracy, there was no steep linear decline in accuracy as lead times increased up to six months, and the greatest value-added over seasonal average baseline models was found for later lead times at four to six months. Higher value-added differences were observed for the second half of the year, but the unusually-early June dengue outbreak in 2022 provided a notable challenge. Spatially, larger errors were found in central and southern provinces, that report higher dengue incidence, alongside contrastingly greater value-added over null baseline models, particularly at shorter lead times. Four contextualised operational utility scenarios were tested through probabilistic classification of outbreak threshold exceedance, with accuracy ranging across probability cut-offs but peaking at 0.83 - 0.94 across scenarios.The value of waiting for the next month’s forecast and utilising different outbreak thresholds was assessed, with heterogeneous results and the distribution of false alarms or missed outbreaks clustering spatially. The results of both the global performance analysis and utility assessments continue to highlight the strong predictive ability of D-MOSS in an operational setting. Lessons should be taken from the higher-than-expected performance over long-term horizons to improve our ability to forecast further into the future, amidst the key insights these results provide into the improvement of operational dengue forecasting for key decision-making situations in Vietnam and beyond.

Figures