The future of forecasting competitions: Design attributes and principles

TL;DR

This paper reviews the design principles of forecasting competitions, analyzing past examples, identifying gaps, and proposing a multi-contest approach to enhance forecasting accuracy and learning.

Contribution

It introduces ten key design attributes for forecasting competitions, maps existing competitions against these attributes, and suggests principles for future contest design, including a multi-contest 'athlon' concept.

Findings

Identified design gaps in past forecasting competitions

Proposed ten attributes for effective competition design

Recommended multi-contest approach to evaluate forecasters

Abstract

Forecasting competitions are the equivalent of laboratory experimentation widely used in physical and life sciences. They provide useful, objective information to improve the theory and practice of forecasting, advancing the field, expanding its usage and enhancing its value to decision and policymakers. We describe ten design attributes to be considered when organizing forecasting competitions, taking into account trade-offs between optimal choices and practical concerns like costs, as well as the time and effort required to participate in them. Consequently, we map all major past competitions in respect to their design attributes, identifying similarities and differences between them, as well as design gaps, and making suggestions about the principles to be included in future competitions, putting a particular emphasis on learning as much as possible from their implementation in order to help improve forecasting accuracy and uncertainty. We discuss that the task of forecasting often presents a multitude of challenges that can be difficult to be captured in a single forecasting contest. To assess the caliber of a forecaster, we, therefore, propose that organizers of future competitions consider a multi-contest approach. We suggest the idea of a forecasting "athlon", where different challenges of varying characteristics take place.