Auditing the Fairness of the US COVID-19 Forecast Hub's Case Prediction Models

TL;DR

This paper evaluates the fairness of COVID-19 forecast models from the US CDC's Forecast Hub, revealing disparities in prediction accuracy across social groups and urbanization levels, highlighting the need for fairness metrics.

Contribution

It provides a comprehensive fairness analysis of COVID-19 forecast models, emphasizing the importance of assessing social equity in predictive modeling.

Findings

Significant performance disparities across racial and ethnic groups.

Higher prediction errors in less urbanized areas.

Encourages reporting of fairness alongside accuracy metrics.

Abstract

The US COVID-19 Forecast Hub, a repository of COVID-19 forecasts from over 50 independent research groups, is used by the Centers for Disease Control and Prevention (CDC) for their official COVID-19 communications. As such, the Forecast Hub is a critical centralized resource to promote transparent decision making. While the Forecast Hub has provided valuable predictions focused on accuracy, there is an opportunity to evaluate model performance across social determinants such as race and urbanization level that have been known to play a role in the COVID-19 pandemic. In this paper, we carry out a comprehensive fairness analysis of the Forecast Hub model predictions and we show statistically significant diverse predictive performance across social determinants, with minority racial and ethnic groups as well as less urbanized areas often associated with higher prediction errors. We hope this work will encourage COVID-19 modelers and the CDC to report fairness metrics together with accuracy, and to reflect on the potential harms of the models on specific social groups and contexts.