Revisiting enumerative two-part crude MDL for Bernoulli and multinomial distributions (Extended version)

TL;DR

This paper compares different MDL coding methods for Bernoulli and multinomial distributions, highlighting the advantages of the enumerative two-part crude MDL code in terms of complexity estimation and compression performance.

Contribution

It introduces a Bayesian interpretation of the enumerative MDL code and demonstrates its superiority over NML in practical model selection and data compression.

Findings

Enumerative MDL code shows better compression than NML.

Strong connection established between enumerative code and NML approach.

Theoretical and experimental results favor using the enumerative code in practice.

Abstract

We leverage the Minimum Description Length (MDL) principle as a model selection technique for Bernoulli distributions and compare several types of MDL codes. We first present a simplistic crude two-part MDL code and a Normalized Maximum Likelihood (NML) code. We then focus on the enumerative two-part crude MDL code, suggest a Bayesian interpretation for finite size data samples, and exhibit a strong connection with the NML approach. We obtain surprising impacts on the estimation of the model complexity together with superior compression performance. This is then generalized to the case of the multinomial distributions. Both the theoretical analysis and the experimental comparisons suggest that one might use the enumerative code rather than NML in practice, for Bernoulli and multinomial distributions.