A Comparison of Aggregation Methods for Probabilistic Forecasts of COVID-19 Mortality in the United States

TL;DR

This study evaluates various aggregation methods for probabilistic COVID-19 mortality forecasts in the U.S., highlighting the effectiveness of median and trimming techniques over simple averaging in different mortality scenarios.

Contribution

It compares and assesses the performance of multiple aggregation methods for probabilistic forecasts without relying on historical accuracy data, providing insights into their relative effectiveness.

Findings

Median and trimming methods outperform simple average in low and medium mortality scenarios.

Simple average performs well in high mortality series.

Aggregation methods can reduce bias from over- or underconfidence in forecasts.

Abstract

The COVID-19 pandemic has placed forecasting models at the forefront of health policy making. Predictions of mortality and hospitalization help governments meet planning and resource allocation challenges. In this paper, we consider the weekly forecasting of the cumulative mortality due to COVID-19 at the national and state level in the U.S. Optimal decision-making requires a forecast of a probability distribution, rather than just a single point forecast. Interval forecasts are also important, as they can support decision making and provide situational awareness. We consider the case where probabilistic forecasts have been provided by multiple forecasting teams, and we aggregate the forecasts to extract the wisdom of the crowd. With only limited information available regarding the historical accuracy of the forecasting teams, we consider aggregation (i.e. combining) methods that do not rely on a record of past accuracy. In this empirical paper, we evaluate the accuracy of aggregation methods that have been previously proposed for interval forecasts and predictions of probability distributions. These include the use of the simple average, the median, and trimming methods, which enable robust estimation and allow the aggregate forecast to reduce the impact of a tendency for the forecasting teams to be under- or overconfident. We use data that has been made publicly available from the COVID-19 Forecast Hub. While the simple average performed well for the high mortality series, we obtained greater accuracy using the median and certain trimming methods for the low and medium mortality series. It will be interesting to see if this remains the case as the pandemic evolves.