Evaluating infectious disease forecasts in a cost-loss situation

TL;DR

This paper introduces a decision-theoretic framework for evaluating epidemiological forecasts by considering costs and losses, demonstrating its application to influenza peak predictions and highlighting its advantages over traditional metrics.

Contribution

It adapts a cost-loss decision-theoretic approach to epidemiological forecast evaluation, accounting for decision-maker costs and losses, which is novel in this context.

Findings

Most models show positive value scores for some cost-loss ratios.

Value scores are sensitive to over- and under-prediction.

No clear link between value scores and traditional forecast rankings.

Abstract

In order for epidemiological forecasts to be useful for decision-makers the forecasts need to be properly validated and evaluated. Although several metrics fore evaluation have been proposed and used none of them account for the potential costs and losses that the decision-maker faces. We have adapted a decision-theoretic framework to an epidemiological context which assigns a Value Score (VS) to each model by comparing the expected expense of the decision-maker when acting on the model forecast to the expected expense obtained from acting on historical event probabilities. The VS depends on the cost-loss ratio and a positive VS implies added value for the decision-maker whereas a negative VS means that historical event probabilities outperform the model forecasts. We apply this framework to a subset of model forecasts of influenza peak intensity from the FluSight Challenge and show that most models exhibit a positive VS for some range of cost-loss ratios. However, there is no clear relationship between the VS and the original ranking of the model forecasts obtained using a modified log score. This is in part explained by the fact that the VS is sensitive to over- vs. under-prediction, which is not the case for standard evaluation metrics. We believe that this type of context-sensitive evaluation will lead to improved utilisation of epidemiological forecasts by decision-makers.