How consistent is my model with the data? Information-Theoretic Model Check

TL;DR

This paper introduces an information-theoretic method to evaluate how well a chosen model class can reproduce observed data, applicable to complex data types like sequential and nonlinear dynamical models.

Contribution

It develops a novel model check based on information theory, providing a two-sided posterior predictive test for assessing model-data consistency.

Findings

Effective in synthetic data scenarios

Applicable to real-world sequential and nonlinear models

Outperforms classical whiteness tests in certain cases

Abstract

The choice of model class is fundamental in statistical learning and system identification, no matter whether the class is derived from physical principles or is a generic black-box. We develop a method to evaluate the specified model class by assessing its capability of reproducing data that is similar to the observed data record. This model check is based on the information-theoretic properties of models viewed as data generators and is applicable to e.g. sequential data and nonlinear dynamical models. The method can be understood as a specific two-sided posterior predictive test. We apply the information-theoretic model check to both synthetic and real data and compare it with a classical whiteness test.