Evaluating Weather Forecasts from a Decision Maker's Perspective

TL;DR

This paper introduces a decision-focused evaluation framework for weather forecasts, demonstrating that traditional forecast accuracy metrics may not reflect their effectiveness in real-world decision-making contexts.

Contribution

It proposes decision calibration as a new method to assess forecast models based on their decision-making utility, revealing discrepancies with traditional forecast accuracy measures.

Findings

Model performance at forecast level does not always predict decision-making success.

Model rankings vary across different decision tasks.

Forecast evaluation should consider decision-specific performance.

Abstract

Standard weather forecast evaluations focus on the forecaster's perspective and on a statistical assessment comparing forecasts and observations. In practice, however, forecasts are used to make decisions, so it seems natural to take the decision-maker's perspective and quantify the value of a forecast by its ability to improve decision-making. Decision calibration provides a novel framework for evaluating forecast performance at the decision level rather than the forecast level. We evaluate decision calibration to compare Machine Learning and classical numerical weather prediction models on various weather-dependent decision tasks. We find that model performance at the forecast level does not reliably translate to performance in downstream decision-making: some performance differences only become apparent at the decision level, and model rankings can change among different decision tasks. Our results confirm that typical forecast evaluations are insufficient for selecting the optimal forecast model for a specific decision task.