Approximation of the Two-Part MDL Code

TL;DR

This paper discusses the challenges and properties of approximating the optimal two-part MDL code for data, highlighting issues like computation time, convergence, and model fit, using Kolmogorov complexity as a measure.

Contribution

It analyzes the theoretical properties of successive approximations to the two-part MDL code, emphasizing their limitations and the role of Kolmogorov complexity.

Findings

Each approximation step may be arbitrarily long to compute.

The sequence of models may not monotonically improve fit.

The optimal model has nearly the best goodness of fit.

Abstract

Approximation of the optimal two-part MDL code for given data, through successive monotonically length-decreasing two-part MDL codes, has the following properties: (i) computation of each step may take arbitrarily long; (ii) we may not know when we reach the optimum, or whether we will reach the optimum at all; (iii) the sequence of models generated may not monotonically improve the goodness of fit; but (iv) the model associated with the optimum has (almost) the best goodness of fit. To express the practically interesting goodness of fit of individual models for individual data sets we have to rely on Kolmogorov complexity.