On the Convergence Speed of MDL Predictions for Bernoulli Sequences

TL;DR

This paper analyzes the convergence speed of MDL-based predictions for Bernoulli sequences, providing new bounds on prediction error and demonstrating near-Bayes performance for certain model classes.

Contribution

It derives a novel upper bound on MDL prediction error for countable Bernoulli classes, showing improved convergence rates over previous exponential bounds.

Findings

Bounded total expected square loss implies convergence with probability one.

New upper bounds on prediction error for countable Bernoulli classes.

Results applicable to classification and hypothesis testing tasks.

Abstract

We consider the Minimum Description Length principle for online sequence prediction. If the underlying model class is discrete, then the total expected square loss is a particularly interesting performance measure: (a) this quantity is bounded, implying convergence with probability one, and (b) it additionally specifies a `rate of convergence'. Generally, for MDL only exponential loss bounds hold, as opposed to the linear bounds for a Bayes mixture. We show that this is even the case if the model class contains only Bernoulli distributions. We derive a new upper bound on the prediction error for countable Bernoulli classes. This implies a small bound (comparable to the one for Bayes mixtures) for certain important model classes. The results apply to many Machine Learning tasks including classification and hypothesis testing. We provide arguments that our theorems generalize to countable classes of i.i.d. models.