The Zoltar forecast archive: a tool to facilitate standardization and storage of interdisciplinary prediction research

TL;DR

The paper introduces Zoltar, a standardized software tool and data model for storing, sharing, and evaluating probabilistic forecasts across disciplines, demonstrated through a COVID-19 forecasting case study.

Contribution

It presents a comprehensive data model for probabilistic predictions and implements it in Zoltar, facilitating standardized storage, access, and evaluation of forecasts.

Findings

Successfully stored and accessed 10^7 forecast rows from 20+ teams

Enabled real-time forecast evaluation and comparison

Promotes reproducibility and standardization in forecasting research

Abstract

Forecasting has emerged as an important component of informed, data-driven decision-making in a wide array of fields. We introduce a new data model for probabilistic predictions that encompasses a wide range of forecasting settings. This framework clearly defines the constituent parts of a probabilistic forecast and proposes one approach for representing these data elements. The data model is implemented in Zoltar, a new software application that stores forecasts using the data model and provides standardized API access to the data. In one real-time case study, an instance of the Zoltar web application was used to store, provide access to, and evaluate real-time forecast data on the order of 10$^7$ rows, provided by over 20 international research teams from academia and industry making forecasts of the COVID-19 outbreak in the US. Tools and data infrastructure for probabilistic forecasts, such as those introduced here, will play an increasingly important role in ensuring that future forecasting research adheres to a strict set of rigorous and reproducible standards.