LOO-PIT: A sensitive posterior test

TL;DR

LOO-PIT is a new statistical tool for astronomy that assesses model fit quality using leave-one-out predictive distributions, providing more sensitive detection of model-data discrepancies than traditional methods.

Contribution

The paper introduces LOO-PIT, a novel posterior predictive test for astrophysics, combining LOO estimation with PIT to improve model fit assessment.

Findings

LOO-PIT and $\\chi^2$ tests detect different signals from contaminants.

Using LOO-PIT and $\\chi^2$ together enhances statistical power.

LOO-PIT outperforms $\\chi^2$ in certain realistic astrophysical cases.

Abstract

With the advent of the next generation of astrophysics experiments, the volume of data available to researchers will be greater than ever. As these projects will significantly drive down statistical uncertainties in measurements, it is crucial to develop novel tools to assess the ability of our models to fit these data within the specified errors. We introduce to astronomy the Leave One Out-Probability Integral Transform (LOO-PIT) technique. This first estimates the LOO posterior predictive distributions based on the model and likelihood distribution specified, then evaluates the quality of the match between the model and data by applying the PIT to each estimated distribution and data point, outputting a LOO-PIT distribution. Deviations between this output distribution and that expected can be characterised visually and with a standard Kolmogorov--Smirnov distribution test. We compare LOO-PIT and the more common $\chi^2$ test using both a simplified model and a more realistic astrophysics problem, where we consider fitting Baryon Acoustic Oscillations in galaxy survey data with contamination from emission line interlopers. LOO-PIT and $\chi^2$ tend to find different signals from the contaminants, and using these tests in conjunction increases the statistical power compared to using either test alone. We also show that LOO-PIT outperforms $\chi^2$ in certain realistic test cases.