Bayesian Meta-Reasoning: Determining Model Adequacy from Within a Small World

TL;DR

This paper introduces a Bayesian approach to evaluate model adequacy within a small world, using a test statistic that guides the search for alternative models without explicit enumeration.

Contribution

It develops a Bayesian framework and a test statistic for assessing model adequacy and guiding model improvement without enumerating alternatives.

Findings

The test statistic effectively tracks model performance across instances.

Asymptotic methods provide an approximate distribution for the test statistic.

Components of the test statistic help identify directions for model refinement.

Abstract

This paper presents a Bayesian framework for assessing the adequacy of a model without the necessity of explicitly enumerating a specific alternate model. A test statistic is developed for tracking the performance of the model across repeated problem instances. Asymptotic methods are used to derive an approximate distribution for the test statistic. When the model is rejected, the individual components of the test statistic can be used to guide search for an alternate model.