"What is a realistic forecast?" Assessing data-driven weather forecasts, a journey from verification to falsification

TL;DR

This paper explores the concept of forecast realism in data-driven weather models, proposing a framework to define and assess it, and emphasizing the importance of falsification alongside traditional verification methods.

Contribution

It introduces a novel framework for defining and assessing forecast realism in machine learning weather models, extending beyond standard verification metrics.

Findings

Identifies three types of forecast realism.

Proposes methodological paths for realism assessment.

Highlights the role of falsification in model evaluation.

Abstract

The artificial intelligence revolution is fueling a paradigm shift in weather forecasting: forecasts are generated with machine learning models trained on large datasets rather than with physics-based numerical models that solve partial differential equations. This new approach proved successful in improving forecast performance as measured with standard verification metrics such as the root mean squared error. At the same time, the realism of data-driven weather forecasts is often questioned and considered as an Achilles' heel of machine learning models. How 'forecast realism' can be defined and how this forecast attribute can be assessed are the two questions simultaneously addressed here. Inspired by the seminal work of Murphy (1993) on the definition of 'forecast goodness', we identify 3 types of realism and discuss methodological paths for their assessment. In this framework, falsification arises as a complementary process to verification and diagnostics when assessing data-driven weather models.