Algorithmic statistics: normal objects and universal models

TL;DR

This paper explores the concept of normal objects in algorithmic statistics, analyzing the relevance of models to data, and demonstrates the existence of diverse normal objects with various model properties.

Contribution

It introduces the notion of normal objects and investigates their properties, including the existence of normal objects with non-strong models and the stability of normality under best fit models.

Findings

Many types of normal strings exist

Existence of a normal object with only non-strong best fitting models

Best fit strong models for normal objects are themselves normal

Abstract

Kolmogorov suggested to measure quality of a statistical hypothesis $P$ for a data $x$ by two parameters: Kolmogorov complexity $C(P)$ of the hypothesis and the probability $P(x)$ of $x$ with respect to $P$. P. G\'acs, J. Tromp, P.M.B. Vit\'anyi discovered a small class of models that are universal in the following sense. Each hypothesis $S_{ij}$ from that class is identified by two integer parameters $i,j$ and for every data $x$ and for each complexity level $\alpha$ there is a hypothesis $S_{ij}$ with $j\le i\le l(x)$ of complexity at most $\alpha$ that has almost the best fit among all hypotheses of complexity at most $\alpha$. The hypothesis $S_{ij}$ is identified by $i$ and the leading $i-j$ bits of the binary representation of the number of strings of complexity at most $i$. On the other hand, the initial data $x$ might be completely irrelevant to the the number of strings of complexity at most $i$. Thus $S_{ij}$ seems to have some information irrelevant to the data, which undermines Kolmogorov's approach: the best hypotheses should not have irrelevant information. To restrict the class of hypotheses for a data $x$ to those that have only relevant information, Vereshchagin introduced a notion of a strong model for $x$: those are models for $x$ whose total conditional complexity conditional to $x$ is negligible. An object $x$ is called normal if for each complexity level $\alpha$ at least one its best fitting model of that complexity is strong. In this paper we show that there are "many types" of normal strings. Our second result states that there is a normal object $x$ such that all its best fitting models $S_{ij}$ are not strong for $x$. Our last result states that every best fit strong model for a normal object is again a normal object.