Consistent Kernel Density Estimation with Non-Vanishing Bandwidth

TL;DR

This paper introduces a fixed-bandwidth kernel density estimator (fbKDE) that remains consistent for estimating continuous densities, challenging the traditional requirement of bandwidth shrinking with sample size, and demonstrates its effectiveness through theoretical analysis and experiments.

Contribution

The paper proposes the fbKDE, a novel fixed-bandwidth estimator that guarantees consistency and provides convergence rates, expanding the scope of kernel density estimation methods.

Findings

fbKDE is consistent for continuous square-integrable densities.

Provides convergence rates for radial and box kernels.

Experimental results favor fbKDE over standard and variable bandwidth KDEs.

Abstract

Consistency of the kernel density estimator requires that the kernel bandwidth tends to zero as the sample size grows. In this paper we investigate the question of whether consistency is possible when the bandwidth is fixed, if we consider a more general class of weighted KDEs. To answer this question in the affirmative, we introduce the fixed-bandwidth KDE (fbKDE), obtained by solving a quadratic program, and prove that it consistently estimates any continuous square-integrable density. We also establish rates of convergence for the fbKDE with radial kernels and the box kernel under appropriate smoothness assumptions. Furthermore, in an experimental study we demonstrate that the fbKDE compares favorably to the standard KDE and the previously proposed variable bandwidth KDE.

Tables

Table 1. Table 1: Performance comparison for different datasets and bandwidth selection methods. For the synthetic datasets we drew 1000 samples, n = 800 𝑛 800 n=800 of which were used for training.

		Rule of Thumb			Cross-Validation
		fbKDE	KDE	vKDE	fbKDE	KDE	vKDE
Bimodal	$J_n^T$	-1.0180	-0.8110	-0.8765	-1.0660	-0.9785	-1.0413
	$\parallel \cdot {\parallel }_{\infty }$	0.2287	1.0141	0.8468	0.1865	0.5534	0.2782
Triangular	$J_n^T$	-1.2889	-1.2897	-1.2958	-1.2332	-1.2095	-1.2073
	$\parallel \cdot {\parallel }_{\infty }$	1.0121	1.0200	1.0923	1.1599	1.1437	1.2242
Trimodal	$J_n^T$	-0.3317	-0.2919	-0.3025	-0.3457	-0.3379	-0.3456
	$\parallel \cdot {\parallel }_{\infty }$	0.2335	0.4571	0.4156	0.1212	0.1879	0.1032
Kurtotic	$J_n^T$	-0.5444	-0.4735	-0.5271	-0.5831	-0.5647	-0.5894
	$\parallel \cdot {\parallel }_{\infty }$	0.2800	0.8122	0.5540	0.1379	0.3902	0.1181
Banana	$J_n^T$	-0.0838	-0.0821	-0.0839	-0.0821	-0.0837	-0.0853
Ringnorm	$J_n^T$	2.4E-09	-2.3E-10	-2.7E-10	-1.7E-10	-3.2E-10	-3.5E-10
Thyroid	$J_n^T$	-0.0932	-0.0448	-0.1415	-0.2765	-0.2514	-0.2083
Diabetes	$J_n^T$	-1.4E-05	-0.0004	-9.8E-04	-0.0010	-0.0007	-0.0010
Waveform	$J_n^T$	1.5E-09	-1.2E-11	1.25E-11	-2.1E-12	-1.2E-11	-1.25E-11
Iris	$J_n^T$	0.0166	-0.0204	0.0058	-0.0102	0.0027	0.0777

Table 2. Table 2: Performance comparison with respect to the sample size for the bimodal density.

		Sample size $n$
		50	250	450	1050	1650	1850	2050
$\parallel \cdot {\parallel }_{\infty }$ error	fbKDE	0.7046	0.5847	0.4836	0.3549	0.1807	0.1642	0.1761
	KDE	1.1341	1.0567	1.1451	1.0833	0.9684	0.9420	0.9160
	vKDE	1.0287	0.8811	1.0459	0.9670	0.8106	0.7562	0.7300
$J_n^T$ error	fbKDE	-0.8985	-0.9623	-0.7487	-1.0788	-0.9787	-1.0493	-0.9722
	KDE	-0.7099	-0.7639	-0.6859	-0.8220	-0.8284	-0.8728	-0.8277
	vKDE	-0.7763	-0.8284	-0.7091	-0.8793	-0.8782	-0.9372	-0.8839

Equations

$$f_{KDE}=\frac{1}{n}\sum _{i=1}^{n}k(\cdot ,X_i),$$

$$f_{\alpha }=\sum _{i=1}^n{\alpha }_{i}k(\cdot ,Z_i)$$

$$J(\alpha )={\int }_{\mathcal{X}}{f}_{\alpha }(x{)}^{2}d\mu (x)-2{\int }_{\mathcal{S}}{f}_{\alpha }(x)f(x)d\mu (x).$$

$$H_n(\alpha ):=\sum _{i=1}^n{\alpha }_i\frac{1}{n-1}\sum _{j\ne i}k(X_j,Z_i)=:\sum _{i=1}^n{\alpha }_i{\hat{h}}_i.$$

$$J_n(\alpha )$$

$${\alpha }^{(n)}:=\underset{\alpha \in A_n}{arg\; min}{J}_n(\alpha ).$$

$$f_{{\alpha }^{(n)}}=\sum _{i=1}^n{{\alpha }^{(n)}}_{i}k(\cdot ,Z_i)$$

$${\Vert f-f_{{\alpha }^{(n)}}\Vert }_2^2\le ϵ+\underset{\alpha \in A_n}{\inf }{\Vert f-f_{\alpha }\Vert }_2^2.$$

$${\text{P}}_{X,\Gamma }\left\{{\Vert f-f_{{\alpha }^{(n)}}\Vert }_2^2>ϵ\right\}\to 0$$

$$f_{\beta }:=\sum _{j=1}^m{\beta }_{j}k(\cdot ,y_j),$$

$${\Vert f-f_{\alpha }\Vert }_2\le {\Vert f-f_{\mathcal{H}}\Vert }_2+{\Vert f_{\mathcal{H}}-f_{\beta }\Vert }_2+{\Vert f_{\beta }-f_{\alpha }\Vert }_2.$$

$${\Vert f-f_{{\alpha }^{(n)}}\Vert }_2^2\lesssim {\left(\frac{1}{n}\right)}^{2/d}{\log }^{1/2}\left(n/\delta \right).$$

$${\Vert f-f_{{\alpha }^{(n)}}\Vert }_2^2\lesssim {\left(\frac{1}{n}\right)}^{C/2}{\log }^{2/d}\left(n/\delta \right).$$

$$J({\alpha }^{(n)})\le {ϵ}_1(n)+J^{\ast }$$

$${\Vert f-\sum _{i=1}^m\frac{c_j{\Vert \lambda \Vert }_1}{m}k(\cdot ,y_j)\Vert }_{\infty }$$

$$\underset{\alpha \in A_n}{\inf }{\Vert f_{\beta }-f_{\alpha }\Vert }_2\le {ϵ}_3(n,m)$$

$${\Vert f-{\tilde{f}}_{{\alpha }^{(n)}}\Vert }_2^2\lesssim \frac{{\log }^{1/2}(n/\delta )}{n^{1/(d+1)}}.$$

$${\Vert f-f_{{\alpha }^{(n)}}\Vert }_2^2\le ϵ+\underset{\alpha \in A_n}{\inf }{\Vert f-f_{\alpha }\Vert }_2^2.$$

$$h_i=h_i(X_i,{\Gamma }_i):={\int }_{\mathcal{S}}k(x,X_i+{\Gamma }_i)f(x)d\mu (x),$$

$${\hat{h}}_i={\hat{h}}_i(X,{\Gamma }_i):=\frac{1}{n-1}\sum _{j\ne i}k(X_j,X_i+{\Gamma }_i),$$

$${\text{P}}_{X,\Gamma }\left\{\underset{\alpha \in A_n}{\sup }\left|H_n(\alpha )-H(\alpha )\right|>ϵ\right\}$$

$$={\text{P}}_{X,\Gamma }\left\{\underset{\alpha \in A_n}{\sup }\left|\sum _{i=1}^n{\alpha }_i({\hat{h}}_i(X,{\Gamma }_i)-h_i(X_i,{\Gamma }_i))\right|>ϵ\right\}$$

$$\le {\text{P}}_{X,\Gamma }\left\{\underset{\alpha \in A_n}{\sup }\sum _{i=1}^n\left|{\alpha }_i\right|\left|{\hat{h}}_i(X,{\Gamma }_i)-h_i(X_i,{\Gamma }_i)\right|>ϵ\right\}$$

$$\le {\text{P}}_{X,\Gamma }\left\{R_n\underset{1\le i\le n}{\max }\left|{\hat{h}}_i(X,{\Gamma }_i)-h_i(X_i,{\Gamma }_i)\right|>ϵ\right\}$$

$$\le \sum _{i=1}^n{\text{P}}_{X,\Gamma }\left\{\left|{\hat{h}}_i(X,{\Gamma }_i)-h_i(X_i,{\Gamma }_i)\right|>\frac{ϵ}{R_n}\right\}.$$

$${\text{P}}_{X,{\Gamma }_i}\left\{(X_1,\dots ,X_n,{\Gamma }_i)\in {\mathcal{A}}_i\right\}$$

$$={\int }_{\mathrm{supp}\left\{f_{\Gamma }\right\}}{\text{P}}_X\left\{(X_1,\dots ,X_n,{\gamma }_i)\in {\mathcal{A}}_i\right\}{f}_{\Gamma }({\gamma }_i)d{\gamma }_i,$$

$${\text{P}}_X\left\{(X_1,\dots ,X_n,{\gamma }_i)\in {\mathcal{A}}_i\right\}$$

$$={\int }_{\mathcal{S}}{\text{P}}_{X_{/i}}\left\{(X_1,\dots ,x_i,\dots ,X_n,{\gamma }_i)\in {\mathcal{A}}_i\right\}f(x_i)d\mu (x_i).$$

$${\text{P}}_{X_{/i}}\left\{(X_{/i},z_i)\in {\mathcal{A}}_i\right\}$$

$$={\text{P}}_{X_{/i}}\left\{\left|\frac{1}{n-1}\sum _{\begin{array}{c}j=1\\ j\ne i\end{array}}^{n}k(X_j,z_i)-\frac{1}{n-1}{\mathbb{E}}_{X_{/i}}\left(\sum _{\begin{array}{c}j=1\\ j\ne i\end{array}}^{n}k(X_j,z_i)\right)\right|>ϵ\right\}$$

$$={\text{P}}_{X_{/i}}\left\{\left|\sum _{\begin{array}{c}j=1\\ j\ne i\end{array}}^{n}k(X_j,z_i)-{\mathbb{E}}_{X_{/i}}\left(\sum _{\begin{array}{c}j=1\\ j\ne i\end{array}}^{n}k(X_j,z_i)\right)\right|>(n-1)ϵ\right\}$$

$$\le 2\text{exp}\left\{-\frac{2(n-1){ϵ}^2}{C_k^2}\right\},$$

$${\text{P}}_{X,\Gamma }\left\{\left|{\hat{h}}_i-h_i\right|>\frac{ϵ}{R_n}\right\}$$

$$\le 2\text{exp}\left\{-\frac{2(n-1){ϵ}^2}{C_k^2{R}_n^2}\right\}\cdot {\int }_{\mathrm{supp}\left\{f_{\Gamma }\right\}}{f}_{\Gamma }({\gamma }_i)d{\gamma }_i\cdot {\int }_{\mathcal{S}}f(x_i)d\mu (x_i)$$

Full text

1 Introduction

Given an iid sample $X_{1},\ldots,X_{n}\in\mathbb{R}^{d}$ drawn according to a probability density $f$, the kernel density estimator is

[TABLE]

where $k$ is a kernel function with parameter $\sigma$. Examples of kernels are functions of the form $k(x,x^{\prime})=\sigma^{-d}g(\sigma^{-2}\left\|x-x^{\prime}\right\|^{2})$, where $\int{\sigma^{-d}g(\sigma^{-2}\left\|x-x^{\prime}\right\|^{2})dx}=1$ for all $x^{\prime}$. A common kernel for density estimation is the Gaussian kernel $k(x,x^{\prime})=(2\pi\sigma^{2})^{-d/2}\exp({-(2\sigma^{2})^{-1}\left\|x-x^{\prime}\right\|^{2}})$. Since its inception by Fix and Hodges [1] and development by Rosenblatt and Parzen [2, 3], the KDE has found numerous applications across a broad range of quantitative fields, and has also been the subject of extensive theoretical investigations, spawning several books (see, e.g, [4, 5, 6, 7]) and hundreds of research articles.

A strength of the KDE is that it makes few assumptions about $f$ and that it is consistent, meaning that it converges to $f$ as $n\to\infty$ [8]. Analysis of the KDE stems from the following application of the triangle inequality in some norm of interest, where $*$ denotes convolution: $\|f_{KDE}-f\|\leq\|f_{KDE}-f*k\|+\|f*k-f\|$. Critical to the analysis of the KDE is the dependence of the bandwidth parameter $\sigma$ on $n$. The first term tends to zero provided $n\sigma^{d}\to\infty$, i.e, the number of data points per unit volume tends to infinity. This is shown using properties of convolutions (since $f_{KDE}$ and $f*k$ may be viewed as convolutions of the kernel with the empirical and true distributions, respectively) and concentration of measure. For the latter term to tend to zero, $\sigma\to 0$ is necessary, so that the kernel behaves like a Dirac measure. With additional assumptions on the smoothness of $f$, the optimal growth of $\sigma$ as a function of $n$ can be determined.

The choice of bandwidth determines the performance of the KDE, and automatically selecting the optimal kernel bandwidth remains a difficult problem. Thus, researchers have developed numerous plug-in rules and cross-validation strategies, all of which are successful in some situations. A recent survey counts no fewer than 30 methods in the literature and cites 6 earlier review papers on the topic [9].

As an alternative to the standard KDE, some authors have investigated weighted KDEs, which have the form $f_{\alpha}=\sum_{i=1}^{n}\alpha_{i}k(\cdot,X_{i}).$ The weight vector $\alpha=(\alpha_{1},\ldots,\alpha_{n})\in\mathbb{R}^{n}$ is learned according to some criterion, and such weighted KDEs have been shown to yield improved performance over the standard KDE in certain situations [10, 11, 12, 13]. Consistency of such estimators has been investigated, but still under the assumption that $\sigma\to 0$ with $n$ [14, 15].

In this work we consider the question of whether it is possible to learn the weights of a weighted KDE such that the resulting density estimator is consistent, for a broad class of $f$, where the bandwidth $\sigma$ remains fixed as $n\to\infty$. This question is of theoretical interest, given that all prior work establishing consistency of a KDE, to our knowledge, requires that the bandwidth shrinks to zero. The question is also of practical interest, since such a density estimator could potentially be less sensitive to the choice of bandwidth than the standard KDE, which, as mentioned above, is the main factor limiting the successful application of KDEs in practice.

In Section 2 below, we introduce a weighted KDE that we refer to as the fixed-bandwidth KDE (fbKDE). Its connection to related work is given in Section 3. The theoretical properties of this estimator, including consistency and rates of convergence with a fixed bandwidth, are established in Section 4. Our analysis relies on reproducing kernel Hilbert spaces, a common structure in machine learning that has seldom been used to understand KDE’s. In Section 5, a simulation study is conducted to compare the fbKDE against the standard KDE and another weighted KDE from the literature. Our results indicate that the fbKDE is a promising alternative to these other methods of density estimation.

2 Fixed bandwidth KDE

We start by assuming access to iid data $X_{1},\dots,X_{n}$ sampled from an unknown distribution with density $f$, having support $\mathcal{S}=\mathop{\mathrm{supp}}\left\{f\right\}$ contained in the known set ${\mathcal{X}}\subset\mathbb{R}^{d}$, and with dominating measure $\mu$. The set ${\mathcal{X}}$ is either compact, in which case $\mu$ is taken to be the Lebesgue measure, or ${\mathcal{X}}=\mathbb{R}^{d}$, in which case $\mu$ is a known finite measure. We study a weighted kernel density estimator of the form

[TABLE]

where $\alpha:=(\alpha_{1},\dots,\alpha_{n})^{T}\in A_{n}\subset\mathbb{R}^{n}$, $Z_{i}=X_{i}+\Gamma_{i}$, and $\Gamma_{i}$ is sampled iid from a known distribution with density $f_{\Gamma}$. Here, $f_{\Gamma}$ is chosen to ensure $Z_{i}\in{\mathcal{X}}$, but not necessarily in $\mathcal{S}$. Note that $f_{\alpha}$ is defined on ${\mathcal{X}}$ and $k$ on ${\mathcal{X}}\times{\mathcal{X}}$. Throughout, $A_{n}$ is taken to be an $\ell_{1}$ ball in $\mathbb{R}^{n}$, that is $A_{n}=\left\{\alpha\in\mathbb{R}^{n}\,\,\,|\,\,\,\left\|\alpha\right\|_{1}\leq R_{n}\right\},$ for $R_{n}\in\mathbb{R}$. The reason for centering the kernels at $Z_{i}=Z_{i}+\Gamma_{i}$ instead of $X_{i}$ is that to accurately approximate $f$ with a fixed bandwidth, we might need centers outside the support of $f$.

To measure the error between $f_{\alpha}$ to $f$ we may consider the $\mathcal{L}^{2}_{\mu}\left(\mathcal{S}\right)$ difference, where $\mathcal{L}^{2}_{\mu}(\mathcal{S})$ is the space of equivalence classes of square integrable functions, and where (using both $f$ for the function and its equivalence class) $\left\|f\right\|_{\mathcal{L}^{2}_{\mu}(\mathcal{S})}^{2}:=\int_{\mathcal{S}}f^{2}d\mu$. However, we do not know the set $\mathcal{S}$ and cannot compute said difference. Hence, we consider the $\mathcal{L}^{2}_{\mu}\left({\mathcal{X}}\right)$ difference from $f_{\alpha}$ to $f$.

To determine the scaling coefficients $\alpha$ we consider minimizing $\left\|f-f_{\alpha}\right\|_{\mathcal{L}^{2}_{\mu}\left({\mathcal{X}}\right)}^{2}$ over $\alpha\in A_{n}$. Since $\left\|f-f_{\alpha}\right\|_{\mathcal{L}^{2}_{\mu}\left({\mathcal{X}}\right)}^{2}=\int_{\mathcal{X}}{f_{\alpha}^{2}}-2\int_{\mathcal{X}}ff_{\alpha}+\int_{\mathcal{X}}{f^{2}}$, the term $\int_{\mathcal{X}}f^{2}$ is independent of $\alpha$, and $f$ is zero outside $S$, we can focus on minimizing

[TABLE]

Define $H(\alpha):=\int_{\mathcal{S}}f_{\alpha}(x)f(x)d\mu(x)=\sum_{i=1}^{n}\alpha_{i}\int_{\mathcal{S}}k(x,Z_{i})f(x)d\mu(x)=:\sum_{i=1}^{n}\alpha_{i}h_{i}$. Since we don’t know $f$, we don’t know the true form of $H(\alpha)$ and $J(\alpha)$. However, the terms $h_{i}$ are expectations with respect to $f$ so we can estimate the term $H(\alpha)$ using the available data $\left\{X_{i}\right\}_{i=1}^{n}$. We use the leave-one-out estimator

[TABLE]

With the aid of $H_{n}(\alpha)$, we define the function

[TABLE]

to which we have access. Let $J^{*}:=\inf_{\alpha\in A_{n}}J(\alpha)$ and

[TABLE]

The estimator

[TABLE]

is referred to as the fixed bandwidth kernel density estimator (fbKDE), and is the subject of our study. In the following sections this estimator is shown to be consistent for a fixed kernel bandwidth $\sigma$ under certain conditions on $f,k,R_{n},$ and $f_{\Gamma}$.

3 Related work

The use of the $\mathcal{L}^{2}$ norm as an objective function for kernel density estimation is not new, and in fact, choosing $\sigma$ to minimize $J_{n}$, with $\alpha_{i}=1/n$, is the so-called least-squares leave-one-out cross-validation technique for bandwidth selection. Minimizing $J_{n}$ subject to the constraints $\alpha_{i}\geq 0$ and $\sum_{i}\alpha_{i}=1$ was proposed by [10], and later rediscovered by [11], who also proposed an efficient procedure for solving the resulting quadratic program, and compared the estimator to the KDE experimentally. This same estimator was later analyzed by [14], who established an oracle inequality and consistency under the usual conditions for consistency of the KDE.

Weighted KDEs have also been developed as a means to enhance the standard KDE in various ways. For example, a weighted KDE was proposed in [12] as a form of multiple kernel learning, where, for every data point, multiple kernels of various bandwidths were assigned, and the weights optimized using $J_{n}$. A robust kernel density estimator was proposed in [13], by viewing the standard KDE as a mean in a function space, and estimating the mean robustly. To improve the computational efficiency of evaluating a KDE, several authors have investigated sparse KDEs, learned by various criteria [16, 17, 18, 19, 20].

The one-class SVM has been shown to converge to a truncated version of $f$ in the $\mathcal{L}^{2}$ norm [21]. If the truncation level (determined by the SVM regularization parameter) is high enough, and the density is bounded, then $f$ is consistently estimated. An ensemble of kernel density estimators is studied by [15], who introduce aggregation procedures such that a weighted combination of standard KDEs of various bandwidths performs as well as the KDE whose bandwidth is chosen by an oracle.

In the above-cited work on weighted KDEs, whenever consistency is shown, it assumes a bandwidth tending to zero. Furthermore, the weights are constrained to be nonnegative. In contrast, we allow the weights on individual kernels to be negative, and this enables our theoretical analysis below. Finally, we remark that the terms “fixed" or “constant" bandwidth have been used previously in the literature to refer to a KDE where each data point is assigned the same bandwidth, as opposed to a “variable bandwidth" KDE where each data point might receive a different bandwidth. We instead use “fixed bandwidth" to mean the bandwidth remains fixed as the sample size grows.

4 Theoretical Properties of the fbKDE

Notation

The space $L_{\nu}^{p}({\mathcal{X}})$ is the set of functions on ${\mathcal{X}}$ for which the $p^{th}$ power of their absolute value is $\nu$-integrable over ${\mathcal{X}}$. $\mathcal{L}^{p}_{\nu}({\mathcal{X}})$ is the set of equivalence classes of $L_{\nu}^{p}({\mathcal{X}})$, where two functions $g_{1}$ and $g_{2}$ are equivalent if $\int_{{\mathcal{X}}}\left\lvert g_{1}-g_{2}\right\rvert^{p}d\nu=0$. The symbol $\left\|\cdot\right\|_{2}$ will denote both the Euclidean norm in $\mathbb{R}^{d}$ as well as the $\mathcal{L}^{2}_{\nu}$ norm; which is used will be clear from the context, as the elements of $\mathbb{R}^{d}$ will be denoted by letters towards the end of the alphabet ($x,y$, and $z$). The set $C({\mathcal{X}})$ denotes the space of continuous functions on ${\mathcal{X}}$. Finally, the support of any function $g$ is denoted by $\mathop{\mathrm{supp}}\left\{g\right\}$.

We call a function $k:{\mathcal{X}}\times{\mathcal{X}}\to\mathbb{R}$ a symmetric positive definite kernel (SPD) [22] if $k$ is symmetric and satisfies the property that for arbitrary $n$ and $\left\{x_{i}\right\}_{i=1}^{n}$ the matrix $[k(x_{i},x_{j})]_{i,j=1}^{n}$ is positive semidefinite. Every SPD kernel has an associated Hilbert space ${\mathcal{H}}_{k}({\mathcal{X}})$ of real-valued functions, called the reproducing kernel Hilbert space (RKHS) of $k$, which we will denote by ${\mathcal{H}}$ when $k$ and ${\mathcal{X}}$ are clear from the context, and for which $k(x,x^{\prime})=\left<k(\cdot,x),k(\cdot,x^{\prime})\right>_{{\mathcal{H}}}$. SPD kernels also exhibit the reproducing property, which states that for any $h\in{\mathcal{H}}$, $h(x)=\left<h,k(\cdot,x)\right>_{{\mathcal{H}}}$. By a radial SPD kernel we mean an SPD kernel $k$ for which there is a strictly monotone function $g$ such that $k(x,x^{\prime})=g(\left\|x-x^{\prime}\right\|_{2})$ for any $x,x^{\prime}\in{\mathcal{X}}$. Note that the Gaussian kernel is a radial SPD kernel.

If $k$ is a radial SPD kernel then $\sup_{x,x^{\prime}\in{\mathcal{X}}}{k(x,x^{\prime})}\leq C_{k}$ for some $C_{k}>0$. This holds because $k(x,x^{\prime})=\left<k(\cdot,x),k(\cdot,x^{\prime})\right>\leq\left\|k(\cdot,x)\right\|\left\|k(\cdot,x^{\prime})\right\|$ by the reproducing property and Cauchy-Schwarz, and $\left\|k(\cdot,x)\right\|\left\|k(\cdot,x^{\prime})\right\|=\sqrt{k(x,x)}\sqrt{k(x^{\prime},x^{\prime})}=g(0)$.

We will make use of the following assumptions:

A0

The set ${\mathcal{X}}$ is either a compact subset of $\mathbb{R}^{d}$, in which case $\mu$ is taken to be the Lebesgue measure, or ${\mathcal{X}}=\mathbb{R}^{d}$, in which case $\mu$ is a known finite measure.

This assumption is always held throughout the paper. It will not be explicitly stated in the statements of results below, but will be remembered in the proofs when needed.

A1

$X_{1},\dots,X_{n}$ are sampled independently and identically distributed (iid) according to $f$. $\Gamma_{1},\dots,\Gamma_{n}$ are sampled iid according to $f_{\Gamma}$. Furthermore $X_{1},\dots,X_{n},\Gamma_{i},\dots,\Gamma_{n}$ are independent.

Given $\left\{(X_{i},\Gamma_{i})\right\}_{i=1}^{n}$ as in A1, we define $Z_{i}:=X_{i}+\Gamma_{i}$ for $1\leq i\leq n$. This notation will be kept throughout the paper.

A2

The kernel $k$ is radial and SPD, with $k(x,x)=C_{k}$ for all $x\in{\mathcal{X}}$. Furthermore, $k$ is Lipschitz, that is, there exists $L_{k}>0$ such that $\left\|k(\cdot,x)-k(\cdot,y)\right\|_{2}\leq L_{k}\left\|x-y\right\|_{2}$ holds for all $x$, $y\in{\mathcal{X}}$.

Recall from eqn. (3) that ${{\alpha}^{(n)}}$ is the minimizer of ${J}_{n}$ over $A_{n}$. To show the consistency of $f_{\alpha}$ the overall approach of the following sections will be to show that $J({{\alpha}^{(n)}})$ is close to $J^{*}=\inf_{\alpha\in A_{n}}\left\|f-f_{\alpha}\right\|_{2}^{2}$ with high probability, and then show that $J(\alpha)$ (and therefore $J^{*}$) can be made arbitrarily small for optimal choice of $\alpha$. We start by stating an oracle inequality relating $J({{\alpha}^{(n)}})$ and $J^{*}$.

Lemma 1**.**

Let $\epsilon>0$. Let $\left\{(X_{i},\Gamma_{i})\right\}_{i=1}^{n}$ satisfy assumption A1. Let $k$ satisfy $\sup_{x,x^{\prime}\in{\mathcal{X}}}{k(x,x^{\prime})}\leq C_{k}$ and $f_{\alpha}$ be as in Equation (1). Let $\delta=2n\exp\left(-\frac{(n-1)\epsilon^{2}}{8C_{k}^{2}R_{n}^{2}}\right)$. With probability $\geq 1-\delta$,

[TABLE]

∎

The proof consists of showing the terms $H_{n}$ of $J_{n}$ concentrate around the terms $H$ of $J$, and is placed in Section 7. This result allows us to focus on the term $J^{*}=\inf_{\alpha\in A_{n}}\left\|f-f_{\alpha}\right\|_{2}^{2}$, which we proceed to do in the following sections.

4.1 Consistency of $f_{{\alpha}^{(n)}}$

We state the consistency of $f_{{{\alpha}^{(n)}}}$ for radial SPD kernels:

Theorem 1**.**

Let $\epsilon>0$. Let $\left\{(X_{i},\Gamma_{i})\right\}_{i=1}^{n}$ satisfy A1, where $f\in{\mathcal{F}}({\mathcal{X}}):=L_{\mu}^{2}({\mathcal{X}})\cap C({\mathcal{X}})$ and $\mathop{\mathrm{supp}}\left\{f_{Z}\right\}\supseteq{\mathcal{X}}$. Let $k$ satisfy A2 and $f_{{{\alpha}^{(n)}}}$ be as in Equation (4). If $A_{n}$ is such that $R_{n}\rightarrow\infty$ but $R_{n}^{2}{\log{n}}/{n}\rightarrow 0$ as $n\rightarrow\infty$, then

[TABLE]

as $n\rightarrow\infty$. ∎

The probability $\text{P}_{X,\Gamma}\left\{\right\}$ is the joint probability of $\left\{(X_{i},\Gamma_{i})\right\}_{i=1}^{n}$. The sketch of the proof is as follows. To analyze the term $J^{*}=\inf_{\alpha\in A_{n}}\left\|f-f_{\alpha}\right\|_{2}^{2}$ from Lemma 1, we use the fact that ${\mathcal{H}}$ is dense in ${\mathcal{F}}$ [22] in the sup-norm sense and that ${\mathcal{H}}^{0}:=\left\{\sum^{N}_{j}c_{j}k(\cdot,y_{j})|y_{j}\in{\mathcal{X}},c_{i}\in\mathbb{R},\text{and}\,\,\,N\in\mathbb{N}\right\}$ is dense in ${\mathcal{H}}$ in the ${\mathcal{H}}-$norm sense [23]. That is, there exists a function $f_{\mathcal{H}}\in{\mathcal{H}}$ arbitrarily close to $f$ and a function $f_{\beta}\in{\mathcal{H}}^{0}$ arbitrarily close to $f_{\mathcal{H}}$. The function $f_{\beta}$ has the form

[TABLE]

where $\beta=(\beta_{1},\dots,\beta_{m})^{T}$. Note this is an abuse of notation since the functions $f_{\alpha}$ and $f_{\beta}$ do not have the same centers nor necessarily the same number of components. By the triangle inequality we have for any $\alpha$ in $A_{n}$:

[TABLE]

By the above denseness results, the first two terms are small. To make the third term small we need two things: that $R_{n}$ is large enough so that there is an $\alpha\in A_{n}$ matching $\beta$, and that there exists centers $\left\{Z_{i}\right\}_{i=1}^{n}$ of $f_{\alpha}$ that are close to the centers $\left\{y_{j}\right\}_{j=1}^{m}$ of $f_{\beta}$, which will be true with a certain probability. In Section 7 we will prove all these approximations and show that the relevant probability is indeed large and approaches one.

4.2 Convergence rates of $f_{{\alpha}^{(n)}}$ for SPD radial kernels

The rates for radial SPD kernels may be slow, since these kernels are "universal" in that they can approximate arbitrary functions in $L^{2}_{\mu}({\mathcal{X}})\cap C({\mathcal{X}})$. To get better rates, we can make stronger assumptions on $f$. Thus, let ${\mathcal{F}}_{k}=\left\{f\,\,\,|\,\,\,f\geq 0\,\,a.e.,\int f(x)d\mu(x)=1,\,\,\,f=\int_{\mathcal{X}}k(\cdot,x)\lambda(x)dx,\,\,\,\lambda\in L^{1}({\mathcal{X}})\right\}$, that is, the space of densities expressible as $k$-smoothed $L_{1}$ functions. Then we obtain the following convergence rates.

Theorem 2**.**

Let $\delta\in(0,1)$. Let $\mathcal{S}={\mathcal{X}}$, $k$ satisfy A2, $\left\{(X_{i},\Gamma_{i})\right\}_{i=1}^{n}$ satisfy A1 with $f\in{\mathcal{F}}_{k}$, and $\min_{z\in{\mathcal{X}}}\left\{f_{Z}(z)\right\}>0$. Let $f_{{\alpha}^{(n)}}$ be as in Equation (4). If $d>4$ and $R_{n}\sim n^{1/2-d/2}$, then with probability $\geq 1-\delta$

[TABLE]

If $d\leq 4$ and $R_{n}\sim n^{(1-C)/2}$ for C an arbitrary constant in $(0,1)$, then with probability $\geq 1-\delta$

[TABLE]

∎

The symbol $\lesssim$ indicates the first term is bounded by a positive constant (independent of $n$ and $d$) times the second term, and $\sim$ means they grow at the same rate. Note the condition $\min_{z\in{\mathcal{X}}}\left\{f_{Z}(z)\right\}>0$ is satisfied, for example, if ${\mathcal{X}}$ is compact and $f_{\Gamma}$ is Gaussian. The first step in proving Theorem 2 is just a reformulation of the oracle inequality from Lemma 1:

Lemma 2**.**

Let $\delta_{1}\in(0,1)$. Let $\left\{(X_{i},\Gamma_{i})\right\}_{i=1}^{n}$ satisfy assumption A1. Let $k$ satisfy assumption A2 and $f_{\alpha}$ be as in Equation (1). Then with probability $\geq 1-\delta_{1}$

[TABLE]

where $\epsilon_{1}(n):=\sqrt{8}C_{k}R_{n}\sqrt{\frac{\log{(4n/\delta_{1})}}{n-1}}$. ∎

Now, for the $J^{*}$ term in Lemma 2 we introduce the function $f_{\beta}$ as in eqn (5) and make the following decomposition, valid for any $\alpha\in A_{n}$: $\left\|f-f_{\alpha}\right\|_{2}\leq\left\|f-f_{\beta}\right\|_{2}+\left\|f_{\beta}-f_{\alpha}\right\|_{2}.$ The following lemma concerns the term $\left\|f-f_{\beta}\right\|_{2}$, and is taken from [24]:

Lemma 3**.**

Let $f\in{\mathcal{F}}_{k}$. For any $m\in\mathbb{N}$ there are m points $\left\{y_{j}\right\}_{j=1}^{m}\subset{\mathcal{X}}$ and $m$ coefficients $\left\{c_{j}\right\}_{j=1}^{m}\subset\mathbb{R}$ such that

[TABLE]

where $\epsilon_{2}(m):=C\left\|\lambda\right\|_{1}\sqrt{\frac{V_{k}}{m}}$ for some absolute constant $C$ and where $V_{k}$ is the VC-dimension of the set $\left\{k(\cdot,x)\,\,\,|\,\,\,x\in{\mathcal{X}}\right\}$. ∎

The VC dimension of a set $\left\{g_{i}\right\}$ is the maximum number of points that can be separated arbitrarily by functions of the form $g_{i}-r$, $r\in\mathbb{R}$. For radial kernels, $V_{k}=d+1$ (see [25], [26]).

Now let $\beta_{j}=\frac{c_{j}\left\|\lambda\right\|_{1}}{m}$ and $f_{\beta}=\sum_{j=1}^{m}\beta_{j}k(\cdot,y_{j})$. For the remaining term $\left\|f_{\beta}-f_{\alpha}\right\|_{2}$ we will proceed as in the proof of Theorem 1. That is, we need that for all $y_{j}$ there is a point $Z_{i_{j}}$ close to it, and then we can approximate $f_{\beta}$ with $f_{\alpha}=\sum_{i=1}^{n}{\alpha_{i}k(\cdot,Z_{i})}$.

Lemma 4**.**

Let $\delta_{2}\in(0,1)$, let $f,m$, and $f_{\beta}$ be as above. Let $\left\{(X_{i},\Gamma_{i})\right\}_{i=1}^{n}$ satisfy assumption A1. With probability $\geq 1-\delta_{2}$

[TABLE]

where $\epsilon_{3}(n,m):=\frac{C}{n^{1/d}}\log^{1/d}{(m/\delta_{2})}$, for $C$ a constant independent of $n$ and $m$. ∎

Putting Lemmas 2, 3, and 4 together and choosing $m=n^{\theta}$ for an appropriate $\theta$ (the exact form is shown in Section 7) we obtain the proof of Theorem 2.

4.3 Convergence rates for the box kernel

While the previous theorem considered radial SPD kernels, the oracle inequality applies more generally, and in this section we present rates for a nonradial kernel, the box kernel. We assume ${\mathcal{X}}=[0,1]^{d}$ and that the kernel centers are predetermined according to a uniform grid. Precise details are given in the proof of Theorem 3. Thus the fbKDE centered at $\left\{y_{i}\right\}_{i=1}^{M}$ is $\tilde{f}_{\alpha}:=\sum_{i=1}^{M}\alpha_{i}k(\cdot,y_{i}),$ where the $\alpha$ weights are learned in the same way as before, the only change being the kernel centers. Let ${\mathcal{F}}=L_{1}({\mathcal{X}})\cap L_{2}({\mathcal{X}})\cap\text{Lip}({\mathcal{X}})$, where $\text{Lip}({\mathcal{X}})$ are the Lipschitz functions on ${\mathcal{X}}$, and let $L_{f}$ be the Lipschitz constant of $f$. Also, let $k$ be the box kernel $k(x,y)=\frac{1}{(2\sigma)^{d}}\bm{1}_{\{\left\|x-y\right\|_{\infty}\leq\sigma\}}$ defined on ${\mathcal{X}}\pm\sigma\times{\mathcal{X}}\pm\sigma$, and for simplicity assume $\sigma=\frac{1}{2q}$ for $q$ a positive integer. The following theorem is proved in Section 7.

Theorem 3**.**

Let $f\in\mathcal{F}$, $\mathop{\mathrm{supp}}\left\{f_{Z}\right\}\supset{\mathcal{X}}$, $\left\{(X_{i},\Gamma_{i})\right\}_{i=1}^{n}$ satisfy A1, and $R_{n}\sim n^{(d-1)/(2d+2)}$. Let $\delta\in\left(0,1\right)$. With probability at least $1-\delta$

[TABLE]

∎

As with the previous results, the stochastic error is controlled via the oracle inequality. Whereas the preceding results leveraged known approximation properties of radial SPD kernels, in the case of the box kernel we give a novel argument for approximating Lipschitz functions with box kernels having a non-vanishing bandwidth.

5 Experimental Results

We now show the performance of the fbKDE as compared to the KDE and variable bandwidth KDE. The variable bandwidth KDE ([27]), which we refer as the vKDE, has the form $f_{vKDE}=\frac{1}{n}\sum_{i=1}^{n}k_{\sigma_{i}}(x,X_{i})$ where each data point has an individual bandwidth $\sigma_{i}$. [27] proposes to use $\sigma_{i}=\sigma\lambda^{1/2}\left(f_{KDE}(X_{i})\right)^{-1/2}$ where $\sigma$ is the standard KDE’s bandwidth parameter and $\lambda$ is the geometric mean of $\left\{f_{KDE}(X_{i})\right\}_{i=1}^{n}$.

When implementing the fbKDE, there are a few considerations. First, when computing $\int_{\mathcal{X}}f_{\alpha}^{2}(x)d\mu(x)$, the first term of $J_{n}(\alpha)$, we must compute the integral $\int_{\mathcal{X}}k(x,z_{i})k(x,z_{j})d\mu(x)$. For computational considerations we assume ${\mathcal{X}}=\mathbb{R}^{d}$ and $\mu$ is the Lebesgue measure. This deviates from our theory, which requires finite $\mu$ for ${\mathcal{X}}=\mathbb{R}^{d}$. Thus, for the Gaussian kernel, which we use in our experiments, this leads to $\int_{\mathcal{X}}k(x,z_{i})k(x,z_{j})d\mu(x)=k_{\sqrt{2}\sigma}(z_{i},z_{j})$. To obtain the $\alpha$ weights we have to solve a quadratic program. We used the ADMM algorithm described in [28], with the aid of the $\ell_{1}$ projection algorithm from [29].

We examine a few benchmark real datasets as well as synthetic data from 1-dimensional densities. The synthetic densities are the triangular density as well as three Gaussian mixtures, a bimodal, a trimodal and a kurtotic, as shown in Figure 2. We computed the parameters $\sigma$, $R_{n}$, and $\sigma_{\gamma}$ in two different ways. First we used rules of thumb. For $\sigma$, we used Silverman’s rule of thumb ([4]). For $R_{n}$ we used, based on the convergence rates, $(n/\log(n))^{1/3}$ for $d\leq 4$ and $(n/\log(n))^{(1/2-2/d)}$ for $d>4$. For $\sigma_{\gamma}$ we used, inspired by the Jaakkola heuristic [30], the median distance to the $5^{th}$ nearest neighbor. Second we used a $V$-fold CV procedure over $100$ parameter combinations drawn randomly from $\Theta_{d}$, with $V=2$ for $n>1000$ and $V=3$ otherwise (see [31]). $\Theta_{d}:=[.1,.5]_{\ell}\times[1.1,2\sqrt{(n)}]_{\ell}\times[.001,.1]_{\ell}$ for $d\leq 4$, and $\Theta_{d}:=[.1,1]_{\ell}\times[1.1,2n^{(1/2-2/d)}]_{\ell}\times[.001,.1]_{\ell}$ for $d>4$, where the subscript $\ell$ indicates logarithmic spacing and where the range is chosen thus since the data is standardized and, for the $R_{n}$ range, informed by the convergence rates. Finally, we used $n$ to be $4/5$ of the original data size for training and the rest for testing. We compute the value $J^{T}_{n}$ as $\int{f_{{\alpha}^{(n)}}}^{2}-2\sum_{i=1}^{n}{{\alpha}^{(n)}}_{i}\frac{1}{n_{T}}\sum_{\ell=1}^{n_{T}}k(x_{\ell}^{(T)},z_{i})$, where $\left\{x_{\ell}^{(T)}\right\}_{\ell=1}^{n_{T}}$ is the test set. Figure 2 shows the bimodal density, the fbKDE, KDE, and vKDE along with the $\alpha$ values for the fbKDE. Table 1 shows the $J_{n}({{\alpha}^{(n)}})^{T}$ error as well as the $\left\|\cdot\right\|_{\infty}$ error, where for some function $g$, $\left\|g\right\|_{\infty}:=\max_{x\in{\mathcal{X}}}\left\lvert g(x)\right\rvert$.

In Figure 2 the density has two Gaussian elements with different widths. It is difficult for KDE and even for the vKDE to approximate such a density. The fbKDE, however, is able to approximate components of different smoothness by making some weights negative. These weights subtract excess mass in regions of refinement. Note that by doing so the fbKDE may itself overshoot and become negative. A similar behavior is exhibited for other densities, in which smoothness varies across regions. In Table 1 we report the performance of the three estimators for both CV and rule of thumb. Note the fbKDE often performs better in terms of the $\left\|\cdot\right\|_{\infty}$, and when the bandwidth is chosen according to a rule of thumb. The fbKDE outperforms both the KDE and vKDE in about half of the cases.

Finally we show, for the bimodal density, a comparison of the performance as the sample size grows. We have chosen the parameters according to the rules of thumb discussed above. Table 2 presents the errors. Note that as the sample size grows the KDE and vKDE do not significantly improve, even though the bandwidth is being updated according to Silverman’s rule. The fbKDE leverages new observations and refines its approximation, and this effect is more obvious for the $\left\|\cdot\right\|_{\infty}$ case. Indeed, the $\left\|\cdot\right\|_{\infty}$ error for fbKDE is smaller at $n=50$ than at for KDE and vKDE at $n=2050$. Similar results hold for the other synthetic datasets. This highlights a notable property of the fbKDE, that it can handle densities with differing degrees of smoothness.

6 Conclusion

We have presented a new member of the family of kernel estimators, the fixed bandwidth kernel density estimator, with desirable statistical properties. In particular, we showed the fbKDE is consistent for fixed kernel parameter $\sigma$, and we provided convergence rates. The fbKDE is a good alternative to the KDE in cases where computing an optimal bandwidth is difficult and for densities that are hard to approximate with inappropriate bandwidths. In these cases and as is shown in the experimental section, the fbKDE can greatly improve on the KDE. The way in which fbKDE achieves a more refined approximation is by balancing properly placed positive and negative weights, sometimes outside of the original support, which is facilitated by the $\Gamma$ variables, and which is not possible with the standard KDE. A few problems of interest remain open. We have illustrated two possible rate of convergence results, but expect many other such results are possible, depending on the choice of kernel and smoothness assumptions. It also remains an open problem to extend our results to more general domains ${\mathcal{X}}$ and to dependent data.

7 Proofs

7.1 Oracle Inequality

First recall the oracle inequality lemma:

Lemma 1

Let $\epsilon>0$. Let $\left\{(X_{i},\Gamma_{i})\right\}_{i=1}^{n}$ satisfy assumption A1. Let $k$ satisfy assumption A2 and $f_{\alpha}$ be as in Equation (1). Let $\delta=2n\exp\left(-\frac{(n-1)\epsilon^{2}}{8C_{k}^{2}R_{n}^{2}}\right)$. With probability $\geq 1-\delta$,

[TABLE]

Proof of Lemma 1.

Recalling the following definition from Section 2, we have

[TABLE]

and

[TABLE]

were we have used the simplified notation $(X,\Gamma_{i})$ to represent $(X_{1},\dots,X_{n},\Gamma_{i})$. To simplify notation further, we let $X_{/i}$ represent $(X_{1},\dots,X_{i-1},X_{i+1},\dots,X_{n})$ and use $\text{P}_{X,\Gamma}\left\{\cdot\right\}$ for $\text{P}_{X_{1},\dots,X_{n},\Gamma_{1},\dots,\Gamma_{n}}\left\{\cdot\right\}$, and the same goes for $\mathbb{E}_{X,\Gamma}\left(\cdot\right)$. We now look at the probability that $H_{n}(\alpha)$ is close to $H(\alpha)$. We have

[TABLE]

Now let ${\mathcal{A}}_{i}:=\left\{(x_{1},\dots,x_{n},\gamma_{i})\in\mathcal{S}^{n}\times\mathop{\mathrm{supp}}\left\{f_{\Gamma}\right\}\mid\left\lvert\widehat{h}_{i}(x,\gamma_{i})-h_{i}(x_{i},\gamma_{i})\right\rvert>\frac{\epsilon}{R_{n}}\right\}$ and note that $\text{P}_{X,\Gamma}\left\{{\mathcal{A}}_{i}\right\}=\text{P}_{X,\Gamma_{i}}\left\{{\mathcal{A}}_{i}\right\}$. We have

[TABLE]

by the independence assumption of A1. Furthermore,

[TABLE]

We now bound the term inside the integral. Since this quantity only depends on $x_{i}$, $\gamma_{i}$ through $x_{i}+\gamma_{i}$, we abbreviate it as $\text{P}_{X_{/i}}\left\{(X_{/i},z_{i})\in{\mathcal{A}}_{i}\right\}$, where $z_{i}=x_{i}+\gamma_{i}$. First, note that $h_{i}(x_{i},\gamma_{i})=\mathbb{E}_{X_{j}}\left(k(X_{j},z_{i})\right)$ for any $j\neq i$. Also, by assumption there is a $C_{k}$ such that $k(x,x^{\prime})\leq C_{k}$ for all $x,x^{\prime}$. Hence,

[TABLE]

where we have used Hoeffding’s inequality. So we obtain

[TABLE]

and

[TABLE]

Therefore, letting $\delta=2n\text{exp}\left\{-\frac{2(n-1)\epsilon^{2}}{C_{k}^{2}R_{n}^{2}}\right\}$, for any $\alpha\in A_{n}$

[TABLE]

So, with probability $\geq 1-\delta$, ${J}_{n}(\alpha)\leq J(\alpha)+2\epsilon$, and $J(\alpha)\leq{J}_{n}(\alpha)+2\epsilon$. Recall that ${J}_{n}({{{\alpha}^{(n)}}})\leq{J}_{n}(\alpha)$ for all $\alpha\in A_{n}$. Then with probability $\geq 1-\delta$,

[TABLE]

If we substitute $\epsilon^{\prime}=4\epsilon$ we obtain the desired result. ∎

7.2 Consistency of $f_{{\alpha}^{(n)}}$

In the following we will make use of the fact that, for continuous positive definite radial kernels, the RKHS norm dominates the $\sup$-norm which in turn dominates the $\mathcal{L}^{2}_{\mu}({\mathcal{X}})$ norm. We state this as a lemma.

Lemma 5**.**

Let $k$, ${\mathcal{X}}$, $\mu$ satisfy assumptions A0 and A2. Then for any $h\in{\mathcal{H}}$ we have

[TABLE]

Proof.

By A0 and A2 $k$ is bounded and continuous, and by Lemma 4.28 of [23], so is every element of ${\mathcal{H}}$. Hence, for $h\in{\mathcal{H}}$ and for ${\mathcal{X}}$ either compact or all of $\mathbb{R}^{d}$ the essential supremum equals the supremum, so we obtain

[TABLE]

where the penultimate inequality follows from the reproducing property and the last inequality is just Cauchy-Schwarz. ∎

Now to prove Theorem 1 we need a couple intermediate lemmas. First, recall

[TABLE]

and

[TABLE]

Note that ${\mathcal{H}}^{0}$ is dense in ${\mathcal{H}}$ ([23]). Assume ${\mathcal{H}}$ is dense in $\mathcal{L}^{2}_{\mu}({\mathcal{X}})$. Then, given $\epsilon>0$ and $\alpha\in A_{n}$, there is an $f_{\mathcal{H}}\in{\mathcal{H}}$ and $f_{\beta}\in{\mathcal{H}}^{0}$ of the form

[TABLE]

such that $\left\|f-f_{\mathcal{H}}\right\|_{2}<\epsilon$ and $\left\|f_{\mathcal{H}}-f_{\beta}\right\|_{{\mathcal{H}}}<\epsilon$. We use these functions to bound $\left\|f-f_{\alpha}\right\|_{2}$:

[TABLE]

and show the last two terms are small in the following lemma.

Lemma 6**.**

Let $\epsilon>0$. Let $k$ satisfy assumption A2 and $f\in L_{\mu}^{2}({\mathcal{X}})\cap C({\mathcal{X}})$. Then

[TABLE]

Proof.

If ${\mathcal{X}}$ is compact, then ${\mathcal{H}}$ is dense in $C({\mathcal{X}})$ (see [22]). Therefore, for fixed $\epsilon$, there is an $f_{\mathcal{H}}\in{\mathcal{H}}$ such that

[TABLE]

and by Lemma 5

[TABLE]

If ${\mathcal{X}}=\mathbb{R}^{d}$, [22] tells us ${\mathcal{H}}$ is dense in $\mathcal{L}^{2}_{\mu}({\mathcal{X}})$, so it directly follows that, for any $\epsilon>0$, there is an $f_{\mathcal{H}}\in{\mathcal{H}}$ satisfying

[TABLE]

Similarly, since ${\mathcal{H}}^{0}$ is dense in ${\mathcal{H}}$ [23], for any fixed $\epsilon$ there is an $f_{\beta}\in{\mathcal{H}}^{0}$ such that

[TABLE]

hence, by lemma 5

[TABLE]

Therefore:

[TABLE]

∎

Note that $f_{\beta}\in{\mathcal{H}}^{0}$ implies $f_{\beta}=\sum_{j=1}^{m}\beta_{j}k(\cdot,y_{j})$ for some $m\in\mathbb{N}$ and where $(\beta_{j},y_{j})\in\mathbb{R}\times{\mathcal{X}}$ for all $1\leq j\leq m$. To make the first term small, we first quantify the continuity of the kernel $k$. Let $\epsilon^{\prime}=\epsilon/\left\|\beta\right\|_{1}$ and define

[TABLE]

where $L(k)$ is the Lipschitz constant of $k$. Then for every $x$ and $y$ in ${\mathcal{X}}$ we have that $\left\|x-y\right\|_{2}\leq\eta_{\epsilon}$ implies $\left\|k(\cdot,x)-k(\cdot,y)\right\|_{2}\leq\mu^{1/2}({\mathcal{X}})\epsilon^{\prime}$.

Recall $f_{\alpha}=\sum_{i=1}^{n}\alpha_{i}k(\cdot,Z_{i})$. With the above result in hand we now have to make sure that at least a subset of the centers $\left\{Z_{i}\right\}_{i=1}^{n}$ of $f_{\alpha}$ are close to the centers $\left\{y_{j}\right\}_{j=1}^{m}$ of $f_{\beta}$ with high probability. First, define $B_{j}=\left\{x\in{\mathcal{X}}\mid\left\|x-y_{j}\right\|_{2}\leq\eta_{\epsilon},\right\}$ and define $P_{Z}:=P_{X,\Gamma}$. Then we obtain the following lemma:

Lemma 7**.**

Let $\epsilon>0$ and $f_{\beta}$, $f_{\alpha}$ and $B_{j}$, $j=1,\dots,m$ as above. Then

[TABLE]

for all $n$, where $p_{0}=\min_{j}{\int_{B_{j}}f_{Z}(z)dz}$.

Proof of Lemma 7.

Let the event $A_{j}^{C}=\left\{\forall z\in\left\{Z_{i}\right\}_{i=1}^{n},\,\,\,z\notin B_{j}\right\}$. Then

[TABLE]

where $p_{j}=\int_{B_{j}}f_{Z}(z)dz$. Let $p_{0}:=\min_{j}\left\{p_{j}\right\}$ and recall that for $p\leq 1$ we have $1-p\leq e^{-p}$. Hence $(1-p_{j})^{n}\leq e^{-np_{j}}\leq e^{-np_{0}}$, and

[TABLE]

For this term to approach zero we need $p_{0}:=\min_{j}{\int_{B_{j}}f_{Z}(z)dz}$ to be strictly positive. This follows from the assumption that $\mathop{\mathrm{supp}}\left(f_{Z}\right)\supseteq{\mathcal{X}}$. Since $m$ and $p_{0}$ only depend on $\epsilon$ and other constants, we get

[TABLE]

as $n\rightarrow\infty$. ∎

Lemma 8**.**

Let ${\mathcal{X}}$, $\mu$ satisfy A0, $\left\{(X_{i},\Gamma_{i})\right\}_{i=1}^{n}$ satisfy A1, and $k$ satisfy A2. Then, $\forall\,$ $\epsilon>0$ $\,\exists\,C$ such that

[TABLE]

for sufficiently large $n$.

Proof.

Let $\delta_{2}=m\exp\left(-np_{0}\right)$. With probability $\geq 1-\delta_{2}$ we have that for every $j$ there is an $i_{j}$ such that $\left\|k(\cdot,z_{i_{j}})-k(\cdot,y_{j})\right\|_{2}\leq\frac{\epsilon}{\left\|\beta\right\|_{1}}$. Then, for $\alpha^{*}$ defined as

[TABLE]

we have

[TABLE]

Note that if for two sequences $\left\{s_{n}\right\},\left\{s^{\prime}_{n}\right\}$ we have $s_{n}\leq s^{\prime}_{n}$ for $n\geq N_{0}$, then $\lim_{n\rightarrow\infty}{s_{n}}\leq\lim_{n\rightarrow\infty}{s^{\prime}_{n}}$, granted such limits exist. Let $s_{n}:=\text{P}_{X,\Gamma}\left\{\inf_{\alpha\in A_{n}}\left\|f-f_{\alpha}\right\|_{2}>3\mu^{1/2}({\mathcal{X}})\epsilon\right\}$ and $s^{\prime}_{n}:=\delta_{2}$. Note that for $n$ large enough, say $n=N_{0}$, $R_{n}\geq\left\|\beta\right\|_{1}$ and therefore $\alpha^{*}\in A_{n}$. So we can see that for $n\geq N_{0}$, the inequality

[TABLE]

holds with probability $\geq 1-\delta_{2}$. That is, for $n\geq N_{0}$, $s_{n}\leq s^{\prime}_{n}$, hence since $s^{\prime}_{n}=\delta_{2}\rightarrow 0$, we get $s_{n}\rightarrow 0$. ∎

Proof of Theorem 1.

[TABLE]

By Lemma 1 the middle term approaches zero and by Lemma 8 the last term does, so

[TABLE]

where $\epsilon^{\prime}=4\epsilon+\left(3\mu^{1/2}({\mathcal{X}})\epsilon\right)^{2}$. ∎

7.3 Convergence Rates of $f_{{\alpha}^{(n)}}$

The proof of Lemma 3 is found in [24]. Now let $\beta_{j}=\frac{c_{j}\left\|\lambda\right\|_{1}}{m}$ and $f_{\beta}=\sum_{j=1}^{m}\beta_{j}k(\cdot,y_{j})$, where $\lambda$, $m$, and $c_{j}$, $y_{j}$, for $j=1,\dots,m$, are as in Lemma P 3. Recall Lemma P 4:

Lemma 4

Let $\delta_{2}\in(0,1)$, let $f\in{\mathcal{F}}_{k}$ and let $f_{\beta}$ and $m$ be as above. Let $\left\{(X_{i},\Gamma_{i})\right\}_{i=1}^{n}$ satisfy assumption A1, then with probability $\geq 1-\delta_{2}$

[TABLE]

where $\epsilon_{3}(n,m):=\frac{C}{n^{1/d}}\log^{1/d}{(m/\delta_{2})}$, for $C$ a constant independent of $n$ and $m$. ∎

Proof of Lemma 4.

Let $\eta=\eta(\epsilon_{3}):=\frac{\epsilon_{3}}{L\left\|\beta\right\|_{1}}$, where $L$ is the Lipschitz constant of the $k$. From Lemmas 7 and 8 we know that with probability $\geq 1-me^{-np_{0}}$ the event that for all $y_{j}$ there is a data point $Z_{i_{j}}$ such that $\left\|y_{j}-Z_{i_{j}}\right\|_{2}\leq\eta$ holds. Recall $p_{0}=\min_{j}\int_{B_{j}}f_{Z}(z)dz$. Now

[TABLE]

where $c^{\prime}$ is the volume of the $d$-dimensional unit ball.

Now pick the $\alpha$ coefficients as in Lemma 7. With probability $\geq 1-me^{-nc^{\prime}\eta^{d}}$ we have:

[TABLE]

Fixing $\delta_{2}\in(0,1)$ and setting $me^{-nc\eta^{d}(\epsilon_{3})}=\delta_{2}$, where $c=c^{\prime}\min_{x\in{\mathcal{X}}}\left\{f^{*}(x)\right\}$, we obtain

[TABLE]

Finally, noting that $\left\|\beta\right\|_{1}=\left\|\lambda\right\|_{1}$, where $\lambda$ is as in Lemma 3, yields the desired result. ∎

We now prove Theorem 2.

Proof of Theorem 2.

We will use the notation and results of Lemmas 2, 3 and 4. First, note that from Lemma 3 we have $\left\|f-f_{\beta}\right\|_{2}\leq\mu^{1/2}({\mathcal{X}})\left\|f-f_{\beta}\right\|_{\infty}\leq\mu^{1/2}({\mathcal{X}})\epsilon_{2}(m).$ So putting the three Lemmas together we have that with probability $\geq 1-(\delta_{1}+\delta_{2})$

[TABLE]

or, taking into account only the dependence on $n,m,\delta_{1}$ and $\delta_{2}$ for the different $\epsilon_{i}$’s, we have that $\left\|f-f_{{{\alpha}^{(n)}}}\right\|_{2}^{2}$ is of the order of

[TABLE]

Now, we want the number $m$ of centers in $f_{\beta}$ close to but no larger than the number $n$ of data points, so we set $m=n^{\theta}$ for some $\theta$ such that $0<\theta<1$. Furthermore, we need $R_{n}$ to grow accordingly, so that $R_{n}=n^{c\theta}$, for $c>0$ a constant possibly dependent on $d$. This yields, ignoring the $\log$ terms for now:

[TABLE]

Setting the first two rates equal we obtain $\theta=1/{2(1+c)}$. Note that if $d>4$ we can match the third term by setting $c=d/4-1$ to obtain an overall rate of $2/d$. Otherwise we can set $c$ to any small number to obtain a rate $1/{2(1+c)}$ slightly slower than $1/2$. Finally, for the $\log$ terms, just let $\delta_{1}=\delta_{2}=\delta/2$ and note that if $d>4$ then $\log^{1/2}$ dominates, and if $d\leq 4$ then $\log^{2/d}$ dominates. ∎

7.4 Box rates

First recall the definition of $\tilde{f}_{\alpha}$ from Section 4.3: For any $M$ and $\epsilon(M)$ let $f_{\beta}=\sum_{i=1}^{M}\beta_{i}k(\cdot,y_{i})$ be such that $\left\|f-f_{\beta}\right\|_{2}<\epsilon(M)$, then

[TABLE]

And, also from section 4.3: ${\mathcal{X}}=[0,1]^{d}$, ${\mathcal{F}}=L_{\mu}^{1}({\mathcal{X}})\cap L_{\mu}^{2}({\mathcal{X}})\cap\text{Lip}({\mathcal{X}})$, where $\text{Lip}({\mathcal{X}})$ are the Lipschitz functions on ${\mathcal{X}}$, $\mu$ is the Lebesgue measure and $L_{f}$ is the Lipschitz constant of $f$. Also, $k$ is the box kernel $k(x,y)=\frac{1}{(2\sigma)^{d}}\bm{1}_{\{\left\|x-y\right\|_{\infty}\leq\sigma\}}$ defined on ${\mathcal{X}}\pm\sigma\times{\mathcal{X}}\pm\sigma$, and for simplicity we assume $\sigma=\frac{1}{2q}$ for $q$ a positive integer.

Theorem 3

Let $f\in\mathcal{F}$, $\mathop{\mathrm{supp}}\left\{f_{Z}\right\}\supset{\mathcal{X}}$, $\left\{(X_{i},\Gamma_{i})\right\}_{i=1}^{n}$ satisfy A1 and $R_{n}\sim n^{(d-1)/(2d+2)}$. Let $\delta\in\left(0,1\right)$, then, with probability greater than or equal to $1-\delta$

[TABLE]

∎

To prove this theorem we begin by reformulating Lemma 1:

Lemma 9**.**

Let $\delta\in(0,1)$. Let $\left\{(X_{i})\right\}_{i=1}^{n}$ satisfy A1 and $k$ satisfy $\sup_{x,x^{\prime}\in{\mathcal{X}}}{k(x,x^{\prime})}\leq C_{k}$. If $\tilde{f}_{{{\alpha}^{(n)}}}$ is as above, then with probability $\geq 1-\delta$

[TABLE]

where $\epsilon_{1}(n):=\sqrt{8}C_{k}R_{M}\sqrt{\frac{\log{(4M/\delta_{1})}}{n-1}}$. ∎

We now take care of the term $\inf_{\alpha\in A_{n}}\left\|f-\tilde{f}_{\alpha}\right\|_{2}$.

Lemma 10**.**

Let $f\in{\mathcal{F}}$. For any $m\in\mathbb{N}$ there is a function $f_{\beta}$ of the form $f_{\beta}=\sum_{i=1}^{(mq)^{d}}\beta_{i}k(\cdot,y_{i})$, where $\left\{y_{i}\right\}_{i=1}^{(mq)^{d}}\subset{\mathcal{X}}\pm\sigma$ and $\left\|\beta\right\|_{1}\leq\left(mq\right)^{d-1}R_{f,\sigma}$ satisfying

[TABLE]

where $\epsilon_{2}(m):=\frac{L_{f}\sqrt{d}}{mq}$. ∎

Proof of Lemma 10.

Let $\iota:=(\iota_{1},\dots,\iota_{d})$ be a multi-index with positive elements and associated index $i$ related by the function $h$:

[TABLE]

and its inverse

[TABLE]

Divide ${\mathcal{X}}=[0,1]^{d}$ into $(mq)^{d}$ hypercube regions of equal volume to form the partition $\left\{\Pi_{\ell=1}^{d}\left[\frac{\iota_{\ell}-1}{mq},\frac{\iota_{\ell}}{mq}\right]\right\}_{\iota_{1},\dots,\iota_{d}=1}^{(mq)}=\left\{T_{i}\right\}_{i=1}^{(mq)^{d}}$, where $T_{i}=\Pi_{\ell=1}^{d}\left[\frac{\iota_{\ell}-1}{mq},\frac{\iota_{\ell}}{mq}\right]$. Now, let

[TABLE]

where $\overline{x}_{i}\in T_{i}$. Any choice of $\overline{x}_{i}$ works but for clarity we choose $\overline{x}_{i}=(\iota_{1}/mq,\cdots,\iota_{d}/mq)$. Note that $f_{m}$ is close to $f$:

[TABLE]

Hence

[TABLE]

Now we note that $f_{m}$ can also be expressed as a sum of fixed bandwidth kernels:

[TABLE]

where

[TABLE]

and $\beta$ is as follows. Let $\beta_{1}=f(\overline{x}_{1})$ and

[TABLE]

for $i\leq 2\leq(mq)^{d}$, where $s_{i}=(\max\left\{1,\iota_{1}-(m-1)\right\},\cdots,\max\left\{1,\iota_{d}-(m-1)\right\})$ and $S_{i}=(\max\left\{1,\iota_{1}-1\right\},\cdots,\max{1,\iota_{d}-1})$ are multi-indices. Note that the $\beta_{i}$’s sequentially capture the residual of the function $f_{m}$ as we travel along the $T_{i}$ regions.

To find $\left\|\beta\right\|_{1}$ note that since $\left\|\overline{x}_{1}-0\right\|_{2}\leq L_{f}\sqrt{d}/(mq)$, we have $\left\lvert\beta_{1}\right\rvert\leq f(0)+L_{f}\sqrt{d}/(mq)$. Also note $\beta_{i}=f(\overline{x}_{i})-f(\overline{x}_{i-1})$ for $2\leq\iota_{1}\leq m$, hence for $2\leq i\leq m$

[TABLE]

For larger $\iota_{1}$ we have $\beta_{i}=f(\overline{x}_{i})-f(\overline{x}_{i-1})+\beta_{i-m}-f(0)\bm{1}_{\{\iota_{\ell}=1\,\,\forall\ell\}}$. Note that when $\iota_{1}=m+1$ we have lost influence of $\beta_{1}$, so

[TABLE]

and, similarly, $\left\lvert\beta_{i}\right\rvert\leq 2L_{f}\frac{\sqrt{d}}{mq}$ for $m+2\leq i\leq 2m$. The process continues such that, in general

[TABLE]

for $i\leq mq$. Adding these we obtain $\frac{q+1}{2}(L_{f}\sqrt{d}+qf_{0})$. Denote this quantity by $e^{\prime}$. Then this process is repeated over every dimension, having $q$ chunks of multiples of $e^{\prime}$, the first multiple is $1m$, the second $2m$, and so on. The final sum is then

[TABLE]

Therefore

[TABLE]

∎

Proof of Theorem 3.

This proof is similar to the proof of Theorem 2. Combining Lemmas 9, 10, setting $R_{n^{\prime}}\sim m^{(d-1)}$, ignoring the $log$ terms for now and considering only the dependence on $n$ and $m$ we obtain

[TABLE]

Setting these terms equal we obtain $m=n^{1/(2d+2)}$, with an overall rate of $n^{-1/(d+1)}$. Adding the log term we obtain

[TABLE]

∎