Uniform reliability tests for forecasting systems with small lead time

TL;DR

This paper introduces uniform reliability tests for forecasting systems with small lead times, addressing the challenge of assessing calibration across entire forecast functions, with rigorous asymptotic distribution results and practical experiments.

Contribution

It develops new reliability tests that are valid for small lead time forecasts, with a universal asymptotic distribution and improved power against various alternatives.

Findings

Tests have a universal asymptotic distribution.

Numerical experiments demonstrate effectiveness on weather data.

Extensions to longer lead times are discussed.

Abstract

A long noted difficulty when assessing the reliability (or calibration) of forecasting systems is that reliability, in general, is a hypothesis not about a finite dimensional parameter but about an entire functional relationship. A calibrated probability forecast for binary events for instance should equal the conditional probability of the event given the forecast for {\em any} value of the forecast. Attempts to estimate deviations from calibration at a specific forecast value meet with the difficulty that the probability of the forecast assuming that value is typically zero. Considering the estimated {\em cumulative} deviations from reliability instead however, tests are presented for which the asymptotic distribution of the test statistic can be established rigorously. The distribution turns out to be universal, provided the forecasts "look one step ahead" only, or in other words, verify at the next time step in the future. Furthermore, the tests develop power against a wide class of alternatives. Numerical experiments for both artificial data as well as operational weather forecasting systems are also presented, as are possible extensions to forecasts with longer lead times.