Recalibrating probabilistic forecasts of epidemics

TL;DR

This paper introduces a recalibration method for probabilistic epidemic forecasts that improves their accuracy and reliability by adjusting forecasts based on past data, applicable as a post-processing step.

Contribution

The authors propose a novel recalibration technique for black-box epidemic forecasts, guaranteeing improved calibration and log score performance, with theoretical and empirical validation.

Findings

Recalibration improves forecast calibration and accuracy.

The method guarantees in-sample calibration improvements.

Application to FluSight Network forecasts shows reliable performance enhancement.

Abstract

Distributional forecasts are important for a wide variety of applications, including forecasting epidemics. Often, forecasts are miscalibrated, or unreliable in assigning uncertainty to future events. We present a recalibration method that can be applied to a black-box forecaster given retrospective forecasts and observations, as well as an extension to make this method more effective in recalibrating epidemic forecasts. This method is guaranteed to improve calibration and log score performance when trained and measured in-sample. We also prove that the increase in expected log score of a recalibrated forecaster is equal to the entropy of the PIT distribution. We apply this recalibration method to the 27 influenza forecasters in the FluSight Network and show that recalibration reliably improves forecast accuracy and calibration. This method is effective, robust, and easy to use as a post-processing tool to improve epidemic forecasts.