Nonparametric Maximum Entropy Probability Density Estimation

TL;DR

This paper introduces a new nonparametric maximum entropy method for estimating univariate probability density functions from sample data, utilizing iterative Monte Carlo optimization and statistical quality assessment.

Contribution

It presents a novel, data-driven approach that does not require prior knowledge, employing ensemble modeling and adaptive algorithms to accurately estimate complex PDFs.

Findings

Estimates converge to true PDFs as sample size increases.

Method effectively handles discontinuities, heavy tails, and singularities.

Provides visual and statistical tools to assess estimation quality.

Abstract

Given a sample of independent and identically distributed random variables, a novel nonparametric maximum entropy method is presented to estimate the underlying continuous univariate probability density function (pdf). Estimates are found by maximizing a log-likelihood function based on single order statistics after transforming through a sequence of trial cumulative distribution functions that iteratively improve using a Monte Carlo random search method. Improvement is quantified by assessing the random variables against the statistical properties of sampled uniform random data. Quality is determined using an empirically derived scoring function that is scaled to be sample size invariant. The scoring function identifies atypical fluctuations, for which threshold values are set to define objective criteria that prevent under-fitting as trial iterations continue to improve the model pdf, and, stopping the iteration cycle before over-fitting occurs. No prior knowledge about the data is required. An ensemble of pdf models is used to reflect uncertainties due to statistical fluctuations in random samples, and the quality of the estimates is visualized using scaled residual quantile plots that show deviations from size-invariant statistics. These considerations result in a tractable method that holistically employs key principles of random variables and their statistical properties combined with employing orthogonal basis functions and data-driven adaptive algorithms. Benchmark tests show that the pdf estimates readily converge to the true pdf as sample size increases. Robust results are demonstrated on several test probability densities that include cases with discontinuities, multi-resolution scales, heavy tails and singularities in the pdf, suggesting a generally applicable approach for statistical inference.