Interpreting Internal Consistency of DES Measurements

TL;DR

This paper investigates the use of Bayesian evidence ratios to assess the internal consistency of DES-Y1 measurements, highlighting how prior choices can bias the interpretation of dataset agreement or tension.

Contribution

It critically examines the dependence of evidence ratios on priors and demonstrates potential biases in assessing dataset consistency in cosmological analyses.

Findings

Evidence ratios are prior dependent and can be biased towards agreement.

The Jeffreys scale may falsely indicate agreement with wide priors.

Simulated analyses reveal how priors can conceal true tensions.

Abstract

Bayesian evidence ratios are widely used to quantify the statistical consistency between different experiments. However, since the evidence ratio is prior dependent, the precise translation between its value and the degree of concordance/discordance requires additional information. The most commonly adopted metric, the Jeffreys scale, can falsely suggest agreement between datasets when priors are chosen to be sufficiently wide. In this work, we examine evidence ratios in a DES-Y1 simulated analysis, focusing on the internal consistency between weak lensing and galaxy clustering. We study two scenarios using simulated data in controlled experiments. First, we calibrate the expected evidence ratio distribution given noise realizations around the best fit DES-Y1 $\Lambda$CDM cosmology. Second, we show the behavior of evidence ratios for noiseless fiducial data vectors simulated using a modified gravity model, which generates internal tension in the $\Lambda$CDM analysis. We show that the choice of prior could conceal the discrepancies between weak lensing and galaxy clustering induced by such models and that the evidence ratio in a DES-Y1 study is, indeed, biased towards agreement.