A Geometric Formulation of Occam's Razor for Inference of Parametric Distributions

TL;DR

This paper introduces a geometric measure called the 'razor' to quantify the complexity of parametric models relative to true distributions, linking Bayesian inference, MDL, and information geometry.

Contribution

It defines the 'razor' as a natural complexity measure, providing a geometric interpretation and corrections to MDL for small data scenarios, and justifies the use of Jeffreys prior.

Findings

Fisher information naturally measures distance on the parameter manifold.

Bayesian posterior converges to the model's razor, linking inference to geometric complexity.

Proposes corrections to MDL for better model selection with limited data.

Abstract

I define a natural measure of the complexity of a parametric distribution relative to a given true distribution called the {\it razor} of a model family. The Minimum Description Length principle (MDL) and Bayesian inference are shown to give empirical approximations of the razor via an analysis that significantly extends existing results on the asymptotics of Bayesian model selection. I treat parametric families as manifolds embedded in the space of distributions and derive a canonical metric and a measure on the parameter manifold by appealing to the classical theory of hypothesis testing. I find that the Fisher information is the natural measure of distance, and give a novel justification for a choice of Jeffreys prior for Bayesian inference. The results of this paper suggest corrections to MDL that can be important for model selection with a small amount of data. These corrections are interpreted as natural measures of the simplicity of a model family. I show that in a certain sense the logarithm of the Bayesian posterior converges to the logarithm of the {\it razor} of a model family as defined here. Close connections with known results on density estimation and ``information geometry'' are discussed as they arise.