Data Consistency Approach to Model Validation

TL;DR

This paper introduces a general, automatic criterion for validating statistical models by assessing their data-generating ability, applicable across various data types and inference problems.

Contribution

It proposes a novel, model-intrinsic data consistency criterion for model validation, covering diverse data types and inference scenarios.

Findings

The criterion effectively evaluates model-data consistency.

It performs well across synthetic and real datasets.

Compared favorably to existing validation methods.

Abstract

In scientific inference problems, the underlying statistical modeling assumptions have a crucial impact on the end results. There exist, however, only a few automatic means for validating these fundamental modelling assumptions. The contribution in this paper is a general criterion to evaluate the consistency of a set of statistical models with respect to observed data. This is achieved by automatically gauging the models' ability to generate data that is similar to the observed data. Importantly, the criterion follows from the model class itself and is therefore directly applicable to a broad range of inference problems with varying data types, ranging from independent univariate data to high-dimensional time-series. The proposed data consistency criterion is illustrated, evaluated and compared to several well-established methods using three synthetic and two real data sets.