Robust Scenario Interpretation from Multi-model Prediction Efforts

TL;DR

This paper develops a method to estimate confidence intervals on the impact of decisions in multi-model projections, such as vaccination effects, using only scenario quantiles without requiring joint probability distributions.

Contribution

It introduces a novel approach to derive confidence intervals on outcome differences from probabilistic projections, even when joint distributions are unavailable or difficult to compute.

Findings

Successfully estimated confidence intervals on vaccination impact metrics.

Validated the approach using US Scenario Modeling Hub data.

Provided a method applicable to various multi-model prediction efforts.

Abstract

Multi-model prediction efforts in infectious disease modeling and climate modeling involve multiple teams independently producing projections under various scenarios. Often these scenarios are produced by the presence and absence of a decision in the future, e.g., no vaccinations (scenario A) vs vaccinations (scenario B) available in the future. The models submit probabilistic projections for each of the scenarios. Obtaining a confidence interval on the impact of the decision (e.g., number of deaths averted) is important for decision making. However, obtaining tight bounds only from the probabilistic projections for the individual scenarios is difficult, as the joint probability is not known. Further, the models may not be able to generate the joint probability distribution due to various reasons including the need to rewrite simulations, and storage and transfer requirements. Without asking the submitting models for additional work, we aim to estimate a non-trivial bound on the outcomes due to the decision variable. We first prove, under a key assumption, that an $\alpha-$confidence interval on the difference of scenario predictions can be obtained given only the quantiles of the predictions. Then we show how to estimate a confidence interval after relaxing that assumption. We use our approach to estimate confidence intervals on reduction in cases, deaths, and hospitalizations due to vaccinations based on model submissions to the US Scenario Modeling Hub.