On fits to correlated and auto-correlated data

TL;DR

This paper addresses the challenge of fitting correlated and auto-correlated data in particle physics, proposing methods to reliably estimate goodness-of-fit without requiring a fully reliable covariance matrix.

Contribution

It introduces robust techniques for estimating goodness-of-fit in correlated and auto-correlated data fits, even when covariance matrices are near-singular.

Findings

Robust p-value estimation methods for correlated data fits

Applicable to lattice QCD and particle physics data analysis

Improved reliability in goodness-of-fit assessments

Abstract

Observables in particle physics and specifically in lattice QCD calculations are often extracted from fits. Standard $\chi^2$ tests require a reliable determination of the covariance matrix and its inverse from correlated and auto-correlated data, a challenging task often leading to close-to-singular estimates. These motivate modifications of the definition of $\chi^2$ such as uncorrelated fits. We show how the goodness-of-fit measured by their p-value can still be estimated robustly for a broad class of such fits.