An Information-Theoretic Approach to Nonparametric Estimation, Model Selection, and Goodness of Fit

TL;DR

This paper introduces an information-theoretic framework for nonparametric density estimation, model selection, and goodness of fit, based on the Optimum Information Principle, providing both theoretical insights and practical algorithms.

Contribution

It develops the optimum information estimator grounded in the Optimum Information Principle, offering a new foundation for density estimation and model evaluation.

Findings

The estimator approximates the true density arbitrarily well.

Provides an absolute criterion for model selection.

Introduces a measure for goodness of fit of parametric models.

Abstract

This paper applies the recently axiomatized Optimum Information Principle (minimize the Kullback-Leibler information subject to all relevant information) to nonparametric density estimation, which provides a theoretical foundation as well as a computational algorithm for maximum entropy density estimation. The estimator, called optimum information estimator, approximates the true density arbitrarily well. As a by-product I obtain a measure of goodness of fit of parametric models (both conditional and unconditional) and an absolute criterion for model selection, as opposed to other conventional methods such as AIC and BIC which are relative measures.