Consistency Tests for Comparing Astrophysical Models and Observations

TL;DR

This paper introduces new statistical consistency tests for astrophysical models that evaluate their agreement with observations, accounting for measurement uncertainties through non-parametric methods and simulations.

Contribution

It develops and applies a set of model-independent consistency tests (ConTESTs) for density and regression models in astronomy, improving model validation practices.

Findings

ConTESTs effectively identify model-observation inconsistencies.

The tests guide model selection and improvement in astrophysics.

Application to real data demonstrates practical utility.

Abstract

In astronomy, there is an opportunity to enhance the practice of validating models through statistical techniques, specifically to account for measurement error uncertainties. While models are commonly used to describe observations, there are instances where there is a lack of agreement between the two. This can occur when models are derived from incomplete theories, when a better-fitting model is not available or when measurement uncertainties are not correctly considered. However, with the application of specific tests that assess the consistency between observations and astrophysical models in a model-independent way, it is possible to address this issue. The consistency tests (ConTESTs) developed in this paper use a combination of non-parametric methods and distance measures to obtain a test statistic that evaluates the closeness of the astrophysical model to the observations. To draw conclusions on the consistency hypothesis, a simulation-based methodology is performed. In particular, we built two tests for density models and two for regression models to be used depending on the case at hand and the power of the test needed. We used ConTEST to examine synthetic examples in order to determine the effectiveness of the tests and provide guidance on using them while building a model. We also applied ConTEST to various astronomy cases, identifying which models were consistent and, if not, identifying the probable causes of rejection.