Assessing model error in counterfactual worlds

TL;DR

This paper explores methods to estimate model error in counterfactual scenarios, emphasizing the importance of model calibration for better decision-making and evaluating different approaches through simulations.

Contribution

It introduces and compares three approaches for estimating model error in counterfactual worlds, offering practical recommendations and discussing scenario design components.

Findings

Model miscalibration significantly impacts scenario accuracy.

Three methods for error estimation have distinct benefits and limitations.

Proper scenario design is crucial for evaluating projection accuracy.

Abstract

Counterfactual scenario modeling exercises that ask "what would happen if?" are one of the most common ways we plan for the future. Despite their ubiquity in planning and decision making, scenario projections are rarely evaluated retrospectively. Differences between projections and observations come from two sources: scenario deviation and model miscalibration. We argue the latter is most important for assessing the value of models in decision making, but requires estimating model error in counterfactual worlds. Here we present and contrast three approaches for estimating this error, and demonstrate the benefits and limitations of each in a simulation experiment. We provide recommendations for the estimation of counterfactual error and discuss the components of scenario design that are required to make scenario projections evaluable.