Extendable MDL

TL;DR

This paper explores how combining the MDL principle with exchangeability leads to using Jeffreys prior, addressing normalization issues and extending applicability within exponential families.

Contribution

It demonstrates the connection between MDL, exchangeability, and Jeffreys prior, including handling cases where Jeffreys prior cannot be normalized.

Findings

Jeffreys prior often used with MDL and exchangeability

Conditions for Jeffreys prior normalization after conditioning

Discussion of exotic cases where normalization fails

Abstract

In this paper we show that combination of the minimum description length principle and a exchange-ability condition leads directly to the use of Jeffreys prior. This approach works in most cases even when Jeffreys prior cannot be normalized. Kraft's inequality links codes and distributions but a closer look at this inequality demonstrates that this link only makes sense when sequences are considered as prefixes of potential longer sequences. For technical reasons only results for exponential families are stated. Results on when Jeffreys prior can be normalized after conditioning on a initializing string are given. An exotic case where no initial string allow Jeffreys prior to be normalized is given and some way of handling such exotic cases are discussed.