A Probabilistic WxChallenge Proposal

TL;DR

This paper proposes three new probabilistic forecasting features for the WxChallenge competition, including automated ensemble-based problems, spread betting, and probability distribution games, evaluated using information gain to improve accessibility and scoring.

Contribution

It introduces novel probabilistic game formats and a scoring method based on information gain, enhancing the WxChallenge with accessible, scientifically sound, and automated probabilistic forecasting elements.

Findings

Automated forecast problems based on ensemble guidance.

Introduction of a spread betting game for confidence allocation.

Use of Shannon information gain for score evaluation.

Abstract

The national forecasting competition WxChallenge, brainchild of Brad Illston at the University of Oklahoma in 2005, has become a cherished institution played across the United States each year. Participants include students, faculty, alumni, and industry professionals. However, forecasts are given as scalar values without expression of uncertainty, probabilities being a keystone of meteorological forecasting today, and previous attempts to add probabilistic elements to WxChallenge have failed partly due to challenges in making probability forecasting accessible to all, and inability to combine scores with different units while also appropriately rewarding forecasts using proper scoring rules. Much of the competition's maintenance relies on dedicated volunteers, highlighting need for more automation. Hence I propose three new features: (1) automated forecast problems based on morning ensemble guidance, forming prediction baselines, thresholds over which the players demonstrate skill in their later forecast; (2) a spread betting game, where the players allocate 100 confidence credits to the over-under for exceeding a percentile (e.g., 50pc) threshold of a variable (e.g., maximum temperature) derived from the ensemble baseline; and (3) a game where players distribute 100 confidence credits across bins of a continuous variable (e.g., accumulated precipitation) approximating a probability mass function. Forecasts are evaluated using Shannon information gained over the baseline forecast, yielding additive units of bits that allow score combinations of different variables and units. Information gain parallels the Brier Score and is likewise a sound measure of skill due its punishment of hedging. This proposal objective is to augment WxChallenge with two new probabilistic games that are accessible, scientifically sound, enjoyable, and optional.