Coverage is not enough: Frequentist tests of simulation-based inference for primordial non-Gaussianity

TL;DR

This paper evaluates the reliability of simulation-based inference (SBI) for cosmological parameters, revealing limitations of coverage diagnostics and highlighting the importance of posterior shape validation beyond standard tests.

Contribution

It compares SBI with likelihood-based inference for primordial non-Gaussianity, demonstrating discrepancies in posterior tails and emphasizing the need for improved validation strategies.

Findings

SBI and LBI agree on posterior means and skewness.

Discrepancies in kurtosis indicate differences in posterior tails.

Higher-order statistics like WST tighten constraints on $f_\mathrm{NL}$.

Abstract

(Abridged) Simulation-based inference (SBI) has emerged as a powerful framework for extracting cosmological information from complex, non-linear data where analytical likelihoods are unavailable. Its reliability is commonly assessed using coverage-based diagnostics under the prior predictive distribution, which probe calibration only in an averaged sense and do not constrain posterior behavior at fixed parameter value, the regime relevant for practical inference. We investigate these limitations in the context of primordial non-Gaussianity, parameterized by $f_\mathrm{NL}$, using simulations of the dark matter halo field. We compare SBI based on contrastive neural ratio estimation (CNRE) with likelihood-based inference (LBI) using the power spectrum, bispectrum, and wavelet scattering transform (WST) coefficients across 1000 realizations. SBI and LBI agree well on posterior means and skewness, while the variance agrees on average but shows weaker realization-by-realization consistency. Larger differences arise in the kurtosis, indicating discrepancies in the posterior tails. These effects are already present for the power spectrum - where the Gaussian likelihood assumed in LBI is best justified - and are most pronounced for the combined power spectrum and bispectrum, where SBI posteriors are often underconfident and can yield weaker constraints than either statistic individually, despite passing coverage tests. WST coefficients further tighten constraints on $f_\mathrm{NL}$, even when restricted to large scales. Our results highlight both the potential of higher-order statistics and the need for validation strategies that probe the posterior shape beyond standard coverage diagnostics.