Beyond Strictly Proper Scoring Rules: The Importance of Being Local
Hailiang Du

TL;DR
This paper emphasizes the importance of local scoring rules, especially the logarithmic score, for evaluating probabilistic forecasts, highlighting issues with nonlocal scores like the Continuous Rank Probability Score and advocating for their inclusion in assessments.
Contribution
The paper clarifies the distinction between local and nonlocal strictly proper scoring rules and advocates for the exclusive use of local scores like the logarithmic score in forecast evaluation.
Findings
Logarithmic score is the only local strictly proper scoring rule.
Nonlocal scores can produce misleading evaluations.
Logarithmic score is invariant under smooth transformations.
Abstract
The evaluation of probabilistic forecasts plays a central role both in the interpretation and in the use of forecast systems and their development. Probabilistic scores (scoring rules) provide statistical measures to assess the quality of probabilistic forecasts. Often, many probabilistic forecast systems are available while evaluations of their performance are not standardized, with different scoring rules being used to measure different aspects of forecast performance. Even when the discussion is restricted to strictly proper scoring rules, there remains considerable variability between them; indeed strictly proper scoring rules need not rank competing forecast systems in the same order when none of these systems are perfect. The locality property is explored to further distinguish scoring rules. The nonlocal strictly proper scoring rules considered are shown to have a property that…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsForecasting Techniques and Applications · Statistical and numerical algorithms · Meteorological Phenomena and Simulations
