Warnings based on risk matrices: a coherent framework with consistent evaluation

TL;DR

This paper presents a comprehensive framework for generating and evaluating warnings from risk matrices using probabilistic forecasts and scoring functions, improving reliability and objectivity in warning systems.

Contribution

It introduces a general, probabilistic framework compatible with CAP and a family of scoring functions for objective evaluation of risk matrix-based warnings.

Findings

Scoring functions effectively evaluate warning performance.

Framework improves warning system reliability.

Synthetic and real-world experiments validate methods.

Abstract

Risk matrices are widely used across a range of fields and have found increasing utility in warning decision practices globally. However, their application in this context presents challenges, which range from potentially perverse warning outcomes to a lack of objective verification (i.e., evaluation) methods. This paper introduces a coherent framework for generating multi-level warnings from risk matrices to address these challenges. The proposed framework is general, is based on probabilistic forecasts of hazard severity or impact and is compatible with the Common Alerting Protocol (CAP). Moreover, it includes a family of consistent scoring functions for objectively evaluating the predictive performance of risk matrix assessments and the warnings they produce. These scoring functions enable the ranking of forecasters or warning systems and the tracking of system improvements by rewarding accurate probabilistic forecasts and compliance with warning service directives. A synthetic experiment demonstrates the efficacy of these scoring functions, while the framework is illustrated through warnings for heavy rainfall based on operational ensemble prediction system forecasts for Tropical Cyclone Jasper (Queensland, Australia, 2023). This work establishes a robust foundation for enhancing the reliability and verifiability of risk-based warning systems.