Nonparametric Estimation and On-Line Prediction for General Stationary Ergodic Sources

TL;DR

This paper introduces a nonparametric, measure-theoretic learning algorithm for online prediction of stationary ergodic sources, effective for both discrete and continuous data without requiring a density function.

Contribution

It presents a novel histogram-based estimator that converges for general stationary ergodic sources, extending nonparametric prediction methods to broader classes of sources.

Findings

Estimator converges for discrete and continuous sources.

No need for probability density functions in continuous case.

Measure-theoretic analysis guarantees convergence of Kullback-Leibler divergence.

Abstract

We proposed a learning algorithm for nonparametric estimation and on-line prediction for general stationary ergodic sources. We prepare histograms each of which estimates the probability as a finite distribution, and mixture them with weights to construct an estimator. The whole analysis is based on measure theory. The estimator works whether the source is discrete or continuous. If it is stationary ergodic, then the measure theoretically given Kullback-Leibler information divided by the sequence length $n$ converges to zero as $n$ goes to infinity. In particular, for continuous sources, the method does not require existence of a probability density function.