Randomized Mixture Models for Probability Density Approximation and Estimation

TL;DR

This paper introduces a randomized mixture model based on RFVL networks for probability density approximation and estimation, with proven convergence and consistency, supported by simulation results.

Contribution

It develops a new randomized mixture model for density estimation using RFVL networks, with a proven EM algorithm for maximum likelihood estimation.

Findings

EM algorithm is globally convergent

Maximum likelihood estimator is consistent

Simulation studies support theoretical results

Abstract

Randomized neural networks (NNs) are an interesting alternative to conventional NNs that are more used for data modeling. The random vector functional-link (RVFL) network is an established and theoretically well-grounded randomized learning model. A key theoretical result for RVFL networks is that they provide universal approximation for continuous maps, on average, almost surely. We specialize and modify this result, and show that RFVL networks can provide functional approximations that converge in Kullback-Leibler divergence, when the target function is a probability density function. Expanding on the approximation results, we demonstrate the the RFVL networks lead to a simple randomized mixture model (MM) construction for density estimation from random data. An expectation-maximization (EM) algorithm is derived for the maximum likelihood estimation of our randomized MM. The EM algorithm is proved to be globally convergent and the maximum likelihood estimator is proved to be consistent. A set of simulation studies is given to provide empirical evidence towards our approximation and density estimation results.