On frequentist confidence intervals in a non-Gaussian regime

TL;DR

This paper compares frequentist confidence interval methods in non-Gaussian MCMC posteriors, revealing their agreement in Gaussian regimes and discrepancies beyond, with implications for cosmological parameter estimation.

Contribution

It provides a systematic comparison of confidence interval methods in non-Gaussian settings and highlights the impact of fixing parameters on interval accuracy.

Findings

Methods agree within 10% in Gaussian regimes

Discrepancies increase beyond Gaussian assumptions

Fixing parameters can underestimate confidence intervals

Abstract

We study frequentist confidence intervals based on graphical profile likelihoods (Wilks' theorem, likelihood integration), and the Feldman-Cousins (FC) prescription, a generalisation of the Neyman belt construction, in a setting with non-Gaussian Markov chain Monte Carlo (MCMC) posteriors. Our simplified setting allows us to recycle the MCMC chain as an input in all methods, including mock simulations underlying the FC approach. We find all methods agree to within $10 \%$ in the close to Gaussian regime, but extending methods beyond their regime of validity leads to greater discrepancies. Importantly, we recover a $\sim 2 \sigma$ shift in cosmological parameters between low and high redshift cosmic chronometer data with the FC method, but only when one fits all parameters back to the mocks. We observe that fixing parameters, a common approach in the literature, risks underestimating confidence intervals.