Surprisingly-early bias in forecasts for unscheduled events

TL;DR

This paper identifies a bias in forecasting unscheduled events caused by early verification, which can distort perceived forecast accuracy, and proposes a method to eliminate this bias by excluding certain early-verified forecasts.

Contribution

It introduces the concept of surprisingly-early bias in forecasts for unscheduled events and offers a correction method to improve forecast evaluation.

Findings

Early verification leads to biased accuracy assessments.

Excluding early-verified forecasts reduces bias.

The bias affects natural catastrophe forecast evaluations.

Abstract

When a dataset contains forecasts on unscheduled events, such as natural catastrophes, outcomes may be censored or ``hidden'' since some events have not yet occurred. This article finds that this can lead to a selection bias which affects the perceived accuracy and calibration of forecasts. This selection bias can be eliminated by excluding forecasts on outcomes which have been verified surprisingly early.