Bayesian jackknife tests with a small number of subsets: Application to HERA 21cm power spectrum upper limits

TL;DR

This paper introduces a Bayesian jackknife test to identify biased data subsets, demonstrating its effectiveness on HERA 21cm power spectrum data and highlighting its broad applicability in various scientific contexts.

Contribution

The paper develops an analytically tractable Bayesian jackknife test for detecting biased subsets in data, with an open source implementation and applications to real and simulated datasets.

Findings

HERA data subsets are generally consistent, supporting combined analysis.

The test effectively identifies biased subsets in simulated and real data.

The method is applicable to diverse fields like CMB and cosmological tensions.

Abstract

We present a Bayesian jackknife test for assessing the probability that a data set contains biased subsets, and, if so, which of the subsets are likely to be biased. The test can be used to assess the presence and likely source of statistical tension between different measurements of the same quantities in an automated manner. Under certain broadly applicable assumptions, the test is analytically tractable. We also provide an open source code, CHIBORG, that performs both analytic and numerical computations of the test on general Gaussian-distributed data. After exploring the information theoretical aspects of the test and its performance with an array of simulations, we apply it to data from the Hydrogen Epoch of Reionization Array (HERA) to assess whether different sub-seasons of observing can justifiably be combined to produce a deeper 21cm power spectrum upper limit. We find that, with a handful of exceptions, the HERA data in question are statistically consistent and this decision is justified. We conclude by pointing out the wide applicability of this test, including to CMB experiments and the $H_0$ tension.