Minimum Encoding Approaches for Predictive Modeling

TL;DR

This paper compares two information-theoretic approaches, MDL and MML, for statistical inference, introduces revised MML estimators, and evaluates their performance on small data sets.

Contribution

It presents two new MML estimators, a pointwise and a volumewise version, improving upon previous MML methods and analyzing their performance relative to MDL.

Findings

MDL yields more accurate predictions on small data sets.

Revised MML estimators outperform original MML methods.

Empirical results favor MDL over MML in certain scenarios.

Abstract

We analyze differences between two information-theoretically motivated approaches to statistical inference and model selection: the Minimum Description Length (MDL) principle, and the Minimum Message Length (MML) principle. Based on this analysis, we present two revised versions of MML: a pointwise estimator which gives the MML-optimal single parameter model, and a volumewise estimator which gives the MML-optimal region in the parameter space. Our empirical results suggest that with small data sets, the MDL approach yields more accurate predictions than the MML estimators. The empirical results also demonstrate that the revised MML estimators introduced here perform better than the original MML estimator suggested by Wallace and Freeman.