Occam's Ghost

TL;DR

This paper introduces a novel regularization method based on minimal bit encoding for non-parametric models, extending Occam's Razor to include model parameters, leading to more efficient probability density estimators.

Contribution

It extends the concept of data encoding to model parameters, providing a true measure of model complexity and enabling automatic feature and parameter pruning.

Findings

Minimizes total bits for better regularization

Prunes irrelevant parameters effectively

Detects features with low probability

Abstract

This article applies the principle of Occam's Razor to non-parametric model building of statistical data, by finding a model with the minimal number of bits, leading to an exceptionally effective regularization method for probability density estimators. The idea comes from the fact that likelihood maximization also minimizes the number of bits required to encode a dataset. However, traditional methods overlook that the optimization of model parameters may also inadvertently play the part in encoding data points. The article shows how to extend the bit counting to the model parameters as well, providing the first true measure of complexity for parametric models. Minimizing the total bit requirement of a model of a dataset favors smaller derivatives, smoother probability density function estimates and most importantly, a phase space with fewer relevant parameters. In fact, it is able prune parameters and detect features with small probability at the same time. It is also shown, how it can be applied to any smooth, non-parametric probability density estimator.