On the Design and use of Ensembles of Multi-model Simulations for Forecasting

TL;DR

This paper examines how to optimally combine multiple probability forecasts in limited-data scenarios, highlighting potential pitfalls of multi-model approaches and analyzing their behavior through a mathematical system.

Contribution

It provides a theoretical analysis of multi-model probability forecast combination, especially under data scarcity, and discusses the limitations of Bayesian Model Averaging in such contexts.

Findings

Multi-model forecasts can be misleading with limited data.

Bayesian Model Averaging identifies the true model as data grows large.

Limitations of multi-model approaches are quantified using a mathematical system.

Abstract

Probability forecasting is common in the geosciences, the finance sector, and elsewhere. It is sometimes the case that one has multiple probability-forecasts for the same target. How is the information in these multiple forecast systems best "combined"? Assuming stationary, then in the limit of a very large forecast-outcome archive, each model-based probability density function can be weighted to form a "multi-model forecast" which will, in expectation, provide the most information. In the case that one of the forecast systems yields a probability distribution which reflects the distribution from which the outcome will be drawn, then Bayesian Model Averaging will identify this model as the number of forecast-outcome pairs goes to infinity. In many applications, like those of seasonal forecasting, data are precious: the archive is often limited to fewer than $2^6$ entries. And no perfect model is in hand. In this case, it is shown that forming a single "multi-model probability forecast" can be expected to prove misleading. These issues are investigated using probability forecasts of a simple mathematical system, which allows most limiting behaviours to be quantified.