Comparison of Combination Methods to Create Calibrated Ensemble Forecasts for Seasonal Influenza in the U.S

TL;DR

This study compares different combination methods for creating calibrated ensemble forecasts of seasonal influenza in the U.S., demonstrating that beta transformation methods outperform traditional linear pooling techniques in predictive accuracy.

Contribution

It introduces and evaluates beta transformation-based ensemble methods, specifically the beta-transformed linear pool and beta mixture model, for influenza forecasting.

Findings

Beta transformation methods outperform traditional linear pools in forecast accuracy.

Ensemble methods with calibration adjustments improve probabilistic scores.

Modest under-prediction observed across methods and seasons.

Abstract

The characteristics of influenza seasons varies substantially from year to year, posing challenges for public health preparation and response. Influenza forecasting is used to inform seasonal outbreak response, which can in turn potentially reduce the societal impact of an epidemic. The United States Centers for Disease Control and Prevention, in collaboration with external researchers, has run an annual prospective influenza forecasting exercise, known as the FluSight challenge. A subset of participating teams has worked together to produce a collaborative multi-model ensemble, the FluSight Network ensemble. Uniting theoretical results from the forecasting literature with domain-specific forecasts from influenza outbreaks, we applied parametric forecast combination methods that simultaneously optimize individual model weights and calibrate the ensemble via a beta transformation. We used the beta-transformed linear pool and the finite beta mixture model to produce ensemble forecasts retrospectively for the 2016/2017 to 2018/2019 influenza seasons in the U.S. We compared their performance to methods currently used in the FluSight challenge, namely the equally weighted linear pool and the linear pool. Ensemble forecasts produced from methods with a beta transformation were shown to outperform those from the equally weighted linear pool and the linear pool for all week-ahead targets across in the test seasons based on average log scores. We observed improvements in overall accuracy despite the beta-transformed linear pool or beta mixture methods' modest under-prediction across all targets and seasons. Combination techniques that explicitly adjust for known calibration issues in linear pooling should be considered to improve ensemble probabilistic scores in outbreak settings.