Direct Ensemble Estimation of Density Functionals

TL;DR

This paper introduces a direct nonparametric method for estimating density functionals using k-NN classifier error rates, and develops an ensemble approach that achieves parametric convergence rates regardless of data dimension.

Contribution

It proposes a novel ensemble technique that improves the convergence rate of density functional estimation without relying on density estimation.

Findings

Ensemble method achieves parametric convergence rates in high dimensions.

The approach bypasses traditional density estimation methods.

Method is effective under certain smoothness conditions.

Abstract

Estimating density functionals of analog sources is an important problem in statistical signal processing and information theory. Traditionally, estimating these quantities requires either making parametric assumptions about the underlying distributions or using non-parametric density estimation followed by integration. In this paper we introduce a direct nonparametric approach which bypasses the need for density estimation by using the error rates of k-NN classifiers asdata-driven basis functions that can be combined to estimate a range of density functionals. However, this method is subject to a non-trivial bias that dramatically slows the rate of convergence in higher dimensions. To overcome this limitation, we develop an ensemble method for estimating the value of the basis function which, under some minor constraints on the smoothness of the underlying distributions, achieves the parametric rate of convergence regardless of data dimension.