Strong Asymptotic Assertions for Discrete MDL in Regression and Classification

TL;DR

This paper establishes strong asymptotic guarantees for the MDL estimator in regression and classification with countable models, demonstrating fast convergence of predictions to the true model under minimal assumptions.

Contribution

It provides finite bounds on Hellinger loss and proves almost sure convergence of the MDL predictor to the true model, extending Solomonoff's universal induction results.

Findings

Finite Hellinger loss bounds under true model assumption

Almost sure convergence of the predictive distribution

Fast convergence rates similar to Solomonoff's theorem

Abstract

We study the properties of the MDL (or maximum penalized complexity) estimator for Regression and Classification, where the underlying model class is countable. We show in particular a finite bound on the Hellinger losses under the only assumption that there is a "true" model contained in the class. This implies almost sure convergence of the predictive distribution to the true one at a fast rate. It corresponds to Solomonoff's central theorem of universal induction, however with a bound that is exponentially larger.