Assessing the global and local uncertainty in scientific evidence in the presence of model misspecification

TL;DR

This paper develops bootstrap methods to quantify uncertainty in scientific evidence estimates under model misspecification, enhancing the reliability of model support assessments in scientific studies.

Contribution

It introduces non-parametric bootstrap techniques for estimating the sampling distribution of evidence measures when models are misspecified, providing global and local confidence intervals.

Findings

Bootstrap methods effectively quantify evidence uncertainty.

Global and local confidence intervals improve inference reliability.

Application to ecology data demonstrates practical utility.

Abstract

Scientists need to compare the support for models based on observed phenomena. The main goal of the evidential paradigm is to quantify the strength of evidence in the data for a reference model relative to an alternative model. This is done via an evidence function, such as $\Delta SIC$, an estimator of the sample size scaled difference of divergences between the generating mechanism and the competing models. To use evidence, either for decision making or as a guide to the accumulation of knowledge, an understanding of the uncertainty in the evidence is needed. This uncertainty is well characterized by the standard statistical theory of estimation. Unfortunately, the standard theory breaks down if the models are misspecified, as it is normally the case in scientific studies. We develop non-parametric bootstrap methodologies for estimating the sampling distribution of the evidence estimator under model misspecification. This sampling distribution allows us to determine how secure we are in our evidential statement. We characterize this uncertainty in the strength of evidence with two different types of confidence intervals, which we term "global" and "local". We discuss how evidence uncertainty can be used to improve scientific inference and illustrate this with a reanalysis of the model identification problem in a prominent landscape ecology study (Grace and Keeley, 2006) using structural equations.