Enhancing the statistical evaluation of earthquake forecasts -- An application to Italy

TL;DR

This paper introduces advanced statistical methods to improve the evaluation of earthquake forecast models, demonstrating their effectiveness on models submitted to the CSEP-Italy experiment and highlighting issues with model calibration.

Contribution

It develops new statistical evaluation techniques supported by mathematical theory, applicable to various forecast types, and enhances existing testing procedures like the T-test.

Findings

Models outperform others in specific scenarios

Significant calibration issues identified in models

Proposed methods are versatile for different forecast distributions

Abstract

Testing earthquake forecasts is essential to obtain scientific information on forecasting models and sufficient credibility for societal usage. We aim at enhancing the testing phase proposed by the Collaboratory for the Study of Earthquake Predictability (CSEP, Schorlemmer et al., 2018) with new statistical methods supported by mathematical theory. To demonstrate their applicability, we evaluate three short-term forecasting models that were submitted to the CSEP-Italy experiment, and two ensemble models thereof. The models produce weekly overlapping forecasts for the expected number of M4+ earthquakes in a collection of grid cells. We compare the models' forecasts using consistent scoring functions for means or expectations, which are widely used and theoretically principled tools for forecast evaluation. We further discuss and demonstrate their connection to CSEP-style earthquake likelihood model testing, and specifically suggest an improvement of the T-test. Then, using tools from isotonic regression, we investigate forecast reliability and apply score decompositions in terms of calibration and discrimination. Our results show where and how models outperform their competitors and reveal a substantial lack of calibration for various models. The proposed methods also apply to full-distribution (e.g., catalog-based) forecasts, without requiring Poisson distributions or making any other type of parametric assumption.