Online Aggregation of Probability Forecasts with Confidence

TL;DR

This paper explores online aggregation of probabilistic forecasts using expert advice, demonstrating the mixability of CRPS and proposing methods to combine specialized experts' predictions with theoretical regret bounds.

Contribution

It establishes that CRPS is a mixable loss function and introduces a scheme for aggregating probabilistic forecasts from specialized experts with overlapping domains.

Findings

CRPS is shown to be a mixable loss function.

A time-independent regret bound for the Vovk algorithm with CRPS is derived.

A smooth specialized experts method effectively combines overlapping probabilistic predictions.

Abstract

The paper presents numerical experiments and some theoretical developments in prediction with expert advice (PEA). One experiment deals with predicting electricity consumption depending on temperature and uses real data. As the pattern of dependence can change with season and time of the day, the domain naturally admits PEA formulation with experts having different ``areas of expertise''. We consider the case where several competing methods produce online predictions in the form of probability distribution functions. The dissimilarity between a probability forecast and an outcome is measured by a loss function (scoring rule). A popular example of scoring rule for continuous outcomes is Continuous Ranked Probability Score (CRPS). In this paper the problem of combining probabilistic forecasts is considered in the PEA framework. We show that CRPS is a mixable loss function and then the time-independent upper bound for the regret of the Vovk aggregating algorithm using CRPS as a loss function can be obtained. Also, we incorporate a ``smooth'' version of the method of specialized experts in this scheme which allows us to combine the probabilistic predictions of the specialized experts with overlapping domains of their competence.