Non-trivial informational closure of a Bayesian hyperparameter

TL;DR

This paper explores the non-trivial informational closure (NTIC) of a Bayesian hyperparameter, analyzing its behavior and relation to information gain within a Markov chain framework, revealing that NTIC can grow indefinitely over time.

Contribution

It explicitly demonstrates the indefinite growth of NTIC for Bayesian hyperparameters and investigates its relation to information gain, providing insights into modeling processes.

Findings

NTIC of the hyperparameter increases indefinitely over time.

One-step pointwise NTIC generally does not indicate information gain.

The study connects NTIC behavior with the interpretability of Bayesian hyperparameters.

Abstract

We investigate the non-trivial informational closure (NTIC) of a Bayesian hyperparameter inferring the underlying distribution of an identically and independently distributed finite random variable. For this we embed both the Bayesian hyper-parameter updating process and the random data process into a Markov chain. The original publication by Bertschinger et al. (2006) mentioned that NTIC may be able to capture an abstract notion of modeling that is agnostic to the specific internal structure of and existence of explicit representations within the modeling process. The Bayesian hyperparameter is of interest since it has a well defined interpretation as a model of the data process and at the same time its dynamics can be specified without reference to this interpretation. On the one hand we show explicitly that the NTIC of the hyperparameter increases indefinitely over time. On the other hand we attempt to establish a connection between a quantity that is a feature of the interpretation of the hyperparameter as a model, namely the information gain, and the one-step pointwise NTIC which is a quantity that does not depend on this interpretation. We find that in general we cannot use the one-step pointwise NTIC as an indicator for information gain. We hope this exploratory work can lead to further rigorous studies of the relation between NTIC and modeling.