When tension is just a fluctuation: How noisy data affect model comparison

TL;DR

This paper investigates how noise in data affects the reliability of model comparison statistics like the Bayesian evidence, especially in cosmology, and proposes methods to quantify and mitigate this noise.

Contribution

It introduces an approximate method to assess the scatter in evidence due to noise and demonstrates how data compression can reduce this effect.

Findings

Scatter in the Bayes factor increases with model tension.

The proposed method accurately estimates evidence distribution with minimal additional computation.

Data compression can significantly suppress noise impact on model comparison.

Abstract

Summary statistics of the likelihood, such as the Bayesian evidence, offer a principled way of comparing models and assessing tension between, or within, the results of physical experiments. Noisy realisations of the data induce scatter in these model comparison statistics. For a realistic case of cosmological inference from large-scale structure we show that the logarithm of the Bayes factor attains scatter of order unity, increasing significantly with stronger tension between the models under comparison. We develop an approximate procedure that quantifies the sampling distribution of the evidence at small additional computational cost and apply it to real data to demonstrate the impact of the scatter, which acts to reduce the significance of any model discrepancies. Data compression is highlighted as a potential avenue to suppressing noise in the evidence to negligible levels, with a proof of concept on Planck cosmic microwave background data.