A general Bayesian model-validation framework based on null-test evidence ratios, with an example application to global 21-cm cosmology

TL;DR

This paper introduces a Bayesian validation framework called BaNTER for comparing composite models, effectively addressing biases from component interactions, demonstrated through an application to global 21-cm cosmology data.

Contribution

The paper presents BaNTER, a novel Bayesian null-test evidence ratio framework, to improve model validation especially when component interactions can bias results.

Findings

BaNTER reliably detects biased inferences in composite models.

Application to 21-cm cosmology validates six models with mock data.

BaNTER complements Bayes factors for unbiased signal recovery.

Abstract

Comparing composite models for multi-component observational data is a prevalent scientific challenge. When fitting composite models, there exists the potential for systematics from a poor fit of one model component to be absorbed by another, resulting in the composite model providing an accurate fit to the data in aggregate but yielding biased a posteriori estimates for individual components. We begin by defining a classification scheme for composite model comparison scenarios, identifying two categories: category I, where models with accurate and predictive components are separable through Bayesian comparison of the unvalidated composite models, and category II, where models with accurate and predictive components may not be separable due to interactions between components, leading to spurious detections or biased signal estimation. To address the limitations of category II model comparisons, we introduce the Bayesian Null Test Evidence Ratio-based (BaNTER) validation framework. Applying this classification scheme and BaNTER to a composite model comparison problem in 21-cm cosmology, where minor systematics from imperfect foreground modelling can bias global 21-cm signal recovery, we validate six composite models using mock data. We show that incorporating BaNTER alongside Bayes-factor-based comparison reliably ensures unbiased inferences of the signal of interest across both categories, positioning BaNTER as a valuable addition to Bayesian inference workflows with potential applications across diverse fields.