The Forecast Trap

TL;DR

This paper highlights the 'forecast trap' phenomenon where selecting models based solely on statistical accuracy can lead to worse real-world outcomes, emphasizing the importance of diverse modeling approaches.

Contribution

It introduces the concept of the forecast trap, demonstrating its occurrence in fisheries management and advocating for broader model sets to prevent it.

Findings

Model accuracy does not always correlate with better outcomes.

Selecting the most statistically accurate models can worsen real-world results.

Using a diverse set of models helps avoid the forecast trap.

Abstract

Encouraged by decision makers' appetite for future information on topics ranging from elections to pandemics, and enabled by the explosion of data and computational methods, model based forecasts have garnered increasing influence on a breadth of decisions in modern society. Using several classic examples from fisheries management, I demonstrate that selecting the model or models that produce the most accurate and precise forecast (measured by statistical scores) can sometimes lead to worse outcomes (measured by real-world objectives). This can create a forecast trap, in which the outcomes such as fish biomass or economic yield decline while the manager becomes increasingly convinced that these actions are consistent with the best models and data available. The forecast trap is not unique to this example, but a fundamental consequence of non-uniqueness of models. Existing practices promoting a broader set of models are the best way to avoid the trap.