Descriptive Dimensionality and Its Characterization of MDL-based Learning and Change Detection

TL;DR

This paper introduces the descriptive dimension (Ddim), an information-theoretic measure for probabilistic models, and demonstrates its fundamental role in governing the convergence, error probabilities, and performance of MDL-based learning and change detection methods.

Contribution

It defines Ddim, relates it to model parameters, and proves its influence on MDL learning convergence and change detection error rates.

Findings

Ddim equals the number of independent parameters for parametric models.

Ddim governs the convergence rate of MDL learning algorithms.

Ddim determines the error probabilities in MDL-based change detection.

Abstract

This paper introduces a new notion of dimensionality of probabilistic models from an information-theoretic view point. We call it the "descriptive dimension"(Ddim). We show that Ddim coincides with the number of independent parameters for the parametric class, and can further be extended to real-valued dimensionality when a number of models are mixed. The paper then derives the rate of convergence of the MDL (Minimum Description Length) learning algorithm which outputs a normalized maximum likelihood (NML) distribution with model of the shortest NML codelength. The paper proves that the rate is governed by Ddim. The paper also derives error probabilities of the MDL-based test for multiple model change detection. It proves that they are also governed by Ddim. Through the analysis, we demonstrate that Ddim is an intrinsic quantity which characterizes the performance of the MDL-based learning and change detection.