A Bayesian quantification of consistency in correlated datasets

TL;DR

This paper introduces three Bayesian methods to assess the internal consistency of correlated datasets, demonstrated on weak lensing data, finding no significant tension.

Contribution

It develops a hierarchical Bayesian framework for consistency testing of correlated datasets, including evidence ratios, parameter difference posteriors, and data domain tests.

Findings

No significant internal tension in KiDS-450 weak lensing data

Bayesian consistency tests can effectively evaluate dataset agreement

Software and data are publicly available for reproducibility

Abstract

We present three tiers of Bayesian consistency tests for the general case of $correlated$ datasets. Building on duplicates of the model parameters assigned to each dataset, these tests range from Bayesian evidence ratios as a global summary statistic, to posterior distributions of model parameter differences, to consistency tests in the data domain derived from posterior predictive distributions. For each test we motivate meaningful threshold criteria for the internal consistency of datasets. Without loss of generality we focus on mutually exclusive, correlated subsets of the same dataset in this work. As an application, we revisit the consistency analysis of the two-point weak lensing shear correlation functions measured from KiDS-450 data. We split this dataset according to large vs. small angular scales, tomographic redshift bin combinations, and estimator type. We do not find any evidence for significant internal tension in the KiDS-450 data, with significances below $3\, \sigma$ in all cases. Software and data used in this analysis can be found at http://kids.strw.leidenuniv.nl/sciencedata.php