Asymptotic Properties of Neural Network Sieve Estimators

TL;DR

This paper investigates the asymptotic behavior of neural network estimators within a nonparametric regression framework, establishing their consistency, convergence rates, and normality, supported by simulation evidence.

Contribution

It extends sieve estimation theory to neural networks, providing the first rigorous asymptotic analysis of neural network estimators in a nonparametric setting.

Findings

Neural network estimators are consistent.

Convergence rates are established.

Asymptotic normality is demonstrated.

Abstract

Neural networks are one of the most popularly used methods in machine learning and artificial intelligence nowadays. Due to the universal approximation theorem (Hornik et al. (1989)), a neural network with one hidden layer can approximate any continuous function on a compact support as long as the number of hidden units is sufficiently large. Statistically, a neural network can be classified into a nonlinear regression framework. However, if we consider it parametrically, due to the unidentifiability of the parameters, it is difficult to derive its asymptotic properties. Instead, we considered the estimation problem in a nonparametric regression framework and use the results from sieve estimation to establish the consistency, the rates of convergence and the asymptotic normality of the neural network estimators. We also illustrate the validity of the theories via simulations.

Tables

Table 1. Table 2: Comparison of errors ‖ f ^ n − f 0 ‖ n 2 superscript subscript norm subscript ^ 𝑓 𝑛 subscript 𝑓 0 𝑛 2 \|\hat{f}_{n}-f_{0}\|_{n}^{2} and the least square errors ℚ n ​ ( f ^ n ) subscript ℚ 𝑛 subscript ^ 𝑓 𝑛 \mathbb{Q}_{n}(\hat{f}_{n}) after 20,000 iterations under different sample sizes.

Sample Sizes	Neural Network			Sine			Piecewise Continuous
	${\Vert {\hat{f}}_n-f_0\Vert }_n^2$	${\mathbb{Q}}_n\left({\hat{f}}_n\right)$		${\Vert {\hat{f}}_n-f_0\Vert }_n^2$	${\mathbb{Q}}_n\left({\hat{f}}_n\right)$		${\Vert {\hat{f}}_n-f_0\Vert }_n^2$	${\mathbb{Q}}_n\left({\hat{f}}_n\right)$
50	3.33E-2	0.519		6.04E-2	0.513		6.20E-1	1.124
100	2.79E-2	0.552		3.04E-2	0.587		3.20E-1	0.920
200	6.05E-3	0.500		1.05E-2	0.501		2.51E-1	0.786
500	8.15E-3	0.484		1.19E-2	0.499		3.26E-1	0.769
1000	3.02E-3	0.475		1.54E-2	0.480		2.98E-2	0.489
2000	2.88E-3	0.500		9.72E-3	0.506		1.69E-2	0.515

Table 2. Table 3: Summary of results from the Shapiro-Wilks test and the Kolmogorov-Smirnov test. We use “NN”, “TRI” and “ND” to denote a neural network described in ( 6.2 ), a trigonometric function described in ( 6.3 ) and a continuous function having a non-differential point described in ( 6.4 ), respectively.

Sample Sizes	Shapiro-Wilks Test				Kolmogorov-Smirnov Test
	NN	TRI	ND		NN	TRI	ND
50	0.878	0.884	0.881		0.584	0.597	0.595
100	0.098	0.095	0.095		0.472	0.508	0.484
200	0.940	0.944	0.944		0.731	0.719	0.708
300	0.884	0.888	0.872		0.976	0.986	0.973
400	0.514	0.525	0.513		0.670	0.754	0.708
500	0.768	0.778	0.768		0.733	0.769	0.733

Equations

$$y_i={\alpha }_0+\sum _{j=1}^r{\alpha }_j\sigma \left({\mathbf{\gamma }}_j^T{\mathbf{x}}_i+{\gamma }_{0,j}\right)+{ϵ}_i,$$

$$\{\mathbf{\theta }:{\gamma }_1=\gamma ,{\alpha }_1=\alpha ,{\gamma }_{0,1}={\gamma }_{0,2}={\alpha }_2={\alpha }_0=0\}\cup$$

$$\{\mathbf{\theta }:{\gamma }_1={\gamma }_2=\gamma ,{\gamma }_{0,1}={\gamma }_{0,2}={\alpha }_0=0,{\alpha }_1+{\alpha }_2=\alpha \}$$

$$\sum _{i=1}^n[f({\mathbf{x}}_i,\mathbf{\theta })-f({\mathbf{x}}_i,{\mathbf{\theta }}^{\prime }){]}^2\to \infty ,\text{ as }n\to \infty ,$$

$$y_i=f_0({\mathbf{x}}_i)+{ϵ}_i,$$

$$Q_n(f)$$

$$=\frac{1}{n}\sum _{i=1}^n(f({\mathbf{x}}_i)-f_0({\mathbf{x}}_i){)}^2+{\sigma }^2.$$

$${\hat{f}}_n={\text{argmin}}_{f\in \mathcal{F}}{\mathbb{Q}}_n(f)={\text{argmin}}_{f\in \mathcal{F}}\frac{1}{n}\sum _{i=1}^n(y_i-f({\mathbf{x}}_i){)}^2.$$

$${\mathcal{F}}_1\subseteq {\mathcal{F}}_2\subseteq \cdots \subseteq {\mathcal{F}}_n\subseteq {\mathcal{F}}_{n+1}\subseteq \cdots \subseteq \mathcal{F},$$

$${\mathbb{Q}}_n({\hat{f}}_n)\le \underset{f\in {\mathcal{F}}_n}{\inf }{\mathbb{Q}}_n(f)+{\mathcal{O}}_p({\eta }_n),$$

$${\mathcal{F}}_{r_n}$$

$$\sum _{j=0}^{r_n}|{\alpha }_j|\le V_n\text{ for some }{V}_n>4\text{ and }\underset{1\le j\le r_n}{\max }\sum _{i=0}^d|{\gamma }_{i,j}|\le M_n\text{ for some }{M}_n>0\},$$

$${\sigma }^{\prime }(z)$$

$$\sigma (z_1)-\sigma (z_2)={\sigma }^{\prime }(\lambda z_1+(1-\lambda )z_2)(z_1-z_2),$$

$$|\sigma (z_1)-\sigma (z_2)|=|{\sigma }^{\prime }(\lambda z_1+(1-\lambda )z_2)||z_1-z_2|\le \frac{1}{4}|z_1-z_2|,$$

$$\underset{f\in {\mathcal{F}}_{r_n}}{\sup }\Vert f{\Vert }_{\infty }\le V_n.$$

$$|f(\mathbf{x})|$$

$$\le |{\alpha }_0|+\sum _{j=1}^{r_n}|{\alpha }_j|\sigma \left({\mathbf{\gamma }}_j^T\mathbf{x}+{\gamma }_{0,j}\right)\le \sum _{j=0}^{r_n}|{\alpha }_j|\le V_n.$$

$$\underset{f\in {\mathcal{F}}_{r_n}}{\sup }\Vert f{\Vert }_{\infty }=\underset{f\in {\mathcal{F}}_{r_n}}{\sup }\underset{\mathbf{x}\in \mathcal{X}}{\sup }|f(\mathbf{x})|\le V_n.$$

$${\mathbb{Q}}_n(f)$$

$$=\frac{1}{n}\sum _{i=1}^n(f_0({\mathbf{x}}_i)+{ϵ}_i-f({\mathbf{x}}_i){)}^2$$

$$=\frac{1}{n}\sum _{i=1}^n(f({\mathbf{x}}_i)-f_0({\mathbf{x}}_i){)}^2-2\frac{1}{n}\sum _{i=1}^n{ϵ}_i(f({\mathbf{x}}_i)-f_0({\mathbf{x}}_i))+\frac{1}{n}\sum _{i=1}^n{ϵ}_i^2.$$

$$H:({\Theta }_n,\Vert \cdot {\Vert }_2)$$

$${\mathbf{\theta }}_n$$

$$\Vert H({\mathbf{\theta }}_{1,n})-H({\mathbf{\theta }}_{2,n}){\Vert }_n^2$$

$$=$$

$$\le$$

$$=$$

$${|{\alpha }_j^{(1)}-{\alpha }_j^{(2)}|\sigma \left({\mathbf{\gamma }}_j^{(2{)}^T}{\mathbf{x}}_i+{\gamma }_{0,j}^{(2)}\right)]}^2$$

$$\le$$

$$\Vert H({\mathbf{\theta }}_{1,n})-H({\mathbf{\theta }}_{2,n}){\Vert }_n<ϵ,$$

$$\underset{f\in {\mathcal{F}}_{r_n}}{\sup }\left|{\mathbb{Q}}_n(f)-Q_n(f)\right|\stackrel{p^{\ast }}{\to }0.$$

$${\mathbb{P}}^{\ast }\left(\underset{f\in {\mathcal{F}}_{r_n}}{\sup }|{\mathbb{Q}}_n(f)-Q_n(f)|>\delta \right)$$

$$=$$

$$\le$$

$$:=$$

$$(I)=\mathbb{P}\left(\left|\frac{1}{n}\sum _{i=1}^n{ϵ}_i^2-{\sigma }^2\right|>\frac{\delta }{2}\right)<\frac{\delta }{2}.$$

Full text

Empirical Processes,

Entropy Integral,

arXiv:0000.0000 \startlocaldefs

\endlocaldefs

,

and

t1To whom correspondence should be addressed.

1 Introduction

With the success of machine learning and artificial intelligence in researches and industry, neural networks have become popularly used methods nowadays. Many newly developed machine learning methods are based on deep neural networks and have achieved great classification and prediction accuracy. We refer interested readers to Goodfellow et al. (2016) for more background and details. In classical statistical learning theory, the consistency and the rate of convergence of the empirical risk minimization principle are of great interest. Many upper bounds have been established for the empirical risk and the sample complexity based on the growth function and the Vapnik-Chervonenkis dimension (see for example, Vapnik (1998); Anthony and Bartlett (2009); Devroye, Györfi and Lugosi (2013)). However, few studies have focused on the asymptotic properties for neural networks. As Thomas J. Sargent said, “artificial intelligence is actually statistics, but in a very gorgeous phrase, it is statistics.” So it is natural and worthwhile to explore whether neural networks possess nice asymptotic properties. As if they do, it may be possible to conduct statistical inference based on neural networks. Throughout this paper, we will focus on the asymptotic properties of neural networks with one hidden layer.

In statistics, fitting a neural network with one hidden layer can be viewed as a parametric nonlinear regression problem:

[TABLE]

where $\epsilon_{1},\ldots,\epsilon_{n}$ are i.i.d. random errors with $\mathbb{E}[\epsilon]=0$ and $\mathbb{E}[\epsilon^{2}]=\sigma^{2}<\infty$ and $\sigma(\cdot)$ is an activation function such as $\sigma(z)=1/(1+e^{-z})$, which is used in this paper. White and Racine (2001) obtained the asymptotic distribution of the resulting estimators under the assumption of the true parameters being unique. In fact, the authors implicitly assumed that the number of hidden units $r$ is known. However, even if we assume that we know the number of hidden units, it is difficult to establish the asymptotic properties for the parameter estimators. In section 6.1, we conducted a simulation based on a single-layer neural network with 2 hidden units. Even for such a simple model, the simulation result suggests that it is unlikely to obtain consistent estimators. Moreover, since the number of hidden units is usually unknown in practice, such an assumption can be easily violated. For example, as pointed out in Fukumizu (1996) and Fukumizu et al. (2003), if the true function is $f_{0}(x)=\alpha\sigma(\gamma x)$, (i.e., the true number of hidden units is 1), and we fit the model using a neural network with two hidden units, then any parameter $\boldsymbol{\theta}=[\alpha_{0},\alpha_{1},\ldots,\alpha_{r},\gamma_{0,1},\ldots,\gamma_{0,r},\boldsymbol{\gamma}_{1}^{T},\ldots,\boldsymbol{\gamma}_{r}^{T}]^{T}$ in the high-dimensional set

[TABLE]

realizes the true function $f_{0}(x)$. Therefore, when the number of hidden units is unknown, the parameters in this parametric nonlinear regression problem are unidentifiable. Theorem 1 in Wu (1981) showed that a necessary condition for the weak consistency of nonlinear least square estimators is

[TABLE]

for all $\boldsymbol{\theta}\neq\boldsymbol{\theta}^{\prime}$ in the parameter space as long as the error distribution has finite Fisher information. Such a condition implies that when the parameters are not identifiable, the resulting nonlinear least squares estimators will be inconsistent, which hinders further explorations on the asymptotic properties for the neural network estimators. Liu and Shao (2003) and Zhu and Zhang (2006) proposed techniques to conduct hypothesis testing under loss of identifiability. However, their theoretical results are not easy to implement in the neural network setting.

Even though a function can have different neural network parametrizations, the function itself can be considered as unique. Moreover, due to the Universal Approximation Theorem (Hornik, Stinchcombe and White, 1989), any continuous function on compact support can be approximated arbitrarily well by a neural network with one hidden layer. So it seems natural to consider it as a nonparametric regression problem and approximate the underlying function class through a class of neural networks with one hidden layer. Specifically, suppose that the true nonparametric regression model is

[TABLE]

where $\epsilon_{1},\ldots,\epsilon_{n}$ are i.i.d. random variables defined on a complete probability space $(\Omega,\mathcal{A},\mathbb{P})$ with $\mathbb{E}[\epsilon]=0$, $\textrm{Var}[\epsilon]=\sigma^{2}<\infty$; $\boldsymbol{x}_{1},\ldots,\boldsymbol{x}_{n}\in\mathcal{X}\subset\mathbb{R}^{d}$ are vectors of covariates with $\mathcal{X}$ being a compact set in $\mathbb{R}^{d}$ and $f_{0}$ is an unknown function needed to be estimated. We assume that $f_{0}\in\mathcal{F}$, where $\mathcal{F}$ is the class of continuous functions with compact supports. Clearly, $f_{0}$ minimizes the population criterion function

[TABLE]

A least squares estimator of the regression function can be obtained by minimizing the empirical squared error loss $\mathbb{Q}_{n}(f)$:

[TABLE]

However, if the class of functions $\mathcal{F}$ is too rich, the resulting least squares estimator may have undesired properties, such as inconsistency (van de Geer, 2000; Shen and Wong, 1994; Shen, 1997). Instead, we can optimize the squared error loss over some less complex function space $\mathcal{F}_{n}$, which is an approximation of $\mathcal{F}$ while the approximation error tends to 0 as the sample size increases. In the language of Grenander (1981), such a sequence of function classes is known as a sieve. More precisely, we consider a sequence of function classes,

[TABLE]

approximating $\mathcal{F}$ in the sense that $\bigcup_{n=1}^{\infty}\mathcal{F}_{n}$ is dense in $\mathcal{F}$. In other words, for each $f\in\mathcal{F}$, there exists $\pi_{n}f\in\mathcal{F}_{n}$ such that $d(f,\pi_{n}f)\to 0$ as $n\to\infty$, where $d(\cdot,\cdot)$ is some pseudo-metric defined on $\mathcal{F}$. With some abuse of notation, an approximate sieve estimator $\hat{f}_{n}$ is defined to be

[TABLE]

where $\eta_{n}\to 0$ as $n\to\infty$.

Throughout the rest of the paper, we focus on the sieve of neural networks with one hidden layer and sigmoid activation function. Specifically, we let

[TABLE]

where $r_{n},V_{n},M_{n}\uparrow\infty$ as $n\to\infty$. Such method has been discussed in previous literatures (e.g. White (1989) and White (1990)). In those papers, consistency of the neural network sieve estimators has been established under random designs. However, there are few results on the asymptotic distribution of the neural network sieve estimators, which will be established in this paper. Throughout this paper, we focus on the fixed design. Hornik, Stinchcombe and White (1989) showed that $\bigcup_{n}\mathcal{F}_{r_{n}}$ is dense in $\mathcal{F}$ under the sup-norm. But when considering the asymptotic properties of the sieve estimators, we use the pseudo-norm $\|f\|_{n}^{2}=n^{-1}\sum_{i=1}^{n}f^{2}(\boldsymbol{x}_{i})$ (see Proposition 7.1 in the Appendix) defined on $\mathcal{F}$ and $\mathcal{F}_{r_{n}}$.

In section 2, we discuss the existence of neural network sieve estimators. The weak consistency and rate of convergence of the neural network sieve estimators will be established in section 3 and section 4, respectively. Section 5 focuses on the asymptotic distribution of the neural network sieve estimators. Simulation results are presented in section 6.

Notations: Throughout the rest of the paper, bold font alphabetic letters and Greek letters are vectors. $C(\mathcal{X})$ is the set of continuous functions defined on $\mathcal{X}$. The symbol $\lesssim$ means “bounded above up to a universal constant” and $a_{n}\sim b_{n}$ means $\frac{a_{n}}{b_{n}}\to 1$ as $n\to\infty$. For a pseudo-metric space $(T,d)$, $N(\epsilon,T,d)$ is its covering number, which is the minimum number of $\epsilon$-balls needed to cover $T$. Its natural logarithm is the entropy number and is denoted by $H(\epsilon,T,d)$.

2 Existence

A natural question to ask is whether the sieve estimator based on neural networks exists. Before addressing this question, we first study some properties of $\mathcal{F}_{r_{n}}$. Proposition 2.1 shows that the sigmoid function is a Lipschitz function with Lipschitz constant $L=1/4$.

Proposition 2.1**.**

A sigmoid function $\sigma(z)=e^{z}/(1+e^{z})$ is a Lipschitz function on $\mathbb{R}$ with Lipschitz constant $1/4$.

Proof.

For all $z_{1},z_{2}\in\mathbb{R}$, $\sigma(z)$ is continuous on $[z_{1},z_{2}]$ and is differentiable on $(z_{1},z_{2})$. Note that

[TABLE]

By using the Mean Value Theorem, we know that

[TABLE]

for some $\lambda\in[0,1]$. Hence

[TABLE]

which means that $\sigma(z)$ is a Lipschitz function on $\mathbb{R}$ with Lipschitz constant $1/4$. ∎

The second proposition provides an upper bound for the envelope function $\sup_{f\in\mathcal{F}_{r_{n}}}|f|$.

Proposition 2.2**.**

For each fixed $n$,

[TABLE]

Proof.

For any $f\in\mathcal{F}_{r_{n}}$ with a fixed $n$ and all $\boldsymbol{x}\in\mathcal{X}$, we have

[TABLE]

Since the right hand side does not depend on $\boldsymbol{x}$ and $f$, we get

[TABLE]

∎

Now we quote a general result from White and Wooldridge (1991). The theorem tells us that under some mild conditions, there exists a sieve approximate estimator and such an estimator is also measurable.

Theorem 2.1** (Theorem 2.2 in White and Wooldridge (1991)).**

Let $(\Omega,\mathcal{A},\mathbb{P})$ be a complete probability space and let $(\Theta,\rho)$ be a pseudo-metric space. Let $\{\Theta_{n}\}$ be a sequence of compact subsets of $\Theta$. Let $\mathbb{Q}_{n}:\Omega\times\Theta_{n}\to\bar{\mathbb{R}}$ be $\mathcal{A}\otimes\mathcal{B}(\Theta_{n})/\mathcal{B}(\bar{\mathbb{R}})$-measurable, and suppose that for each $\omega\in\Omega$, $\mathbb{Q}_{n}(\omega,\cdot)$ is lower semicontinuous on $\Theta_{n}$, $n=1,2,\ldots$. Then for each $n=1,2,\ldots$, there exists $\hat{\theta}_{n}:\Omega\to\Theta_{n}$, $\mathcal{A}/\mathcal{B}(\Theta_{n})$-measurable such that for each $\omega\in\Omega$, $\mathbb{Q}_{n}(\omega,\hat{\theta}_{n}(\omega))=\inf_{\theta\in\Theta_{n}}\mathbb{Q}_{n}(\omega,\theta)$.

Note that

[TABLE]

Since the randomness only comes from $\epsilon_{i}$’s, it is clear that $\mathbb{Q}_{n}$ is a measurable function and for a fixed $\omega$, $\mathbb{Q}_{n}$ is continuous in $f$. Therefore, to show the existence of the sieve estimator, it suffices to show that $\mathcal{F}_{r_{n}}$ is compact in $C(\mathcal{X})$, which is proved in the following lemma.

Lemma 2.1**.**

Let $\mathcal{X}$ be a compact subset of $\mathbb{R}^{d}$. Then for each fixed $n$, $\mathcal{F}_{r_{n}}$ is a compact set.

Proof.

For each fixed $n$, let $\boldsymbol{\theta}_{n}=[\alpha_{0},\ldots,\alpha_{r_{n}},\boldsymbol{\gamma}_{0,1},\ldots,\gamma_{0,r_{n}},\boldsymbol{\gamma}_{1}^{T},\ldots,\boldsymbol{\gamma}_{r_{n}}^{T}]^{T}$ belong to $[-V_{n},V_{n}]^{r_{n}+1}\times[-M_{n},M_{n}]^{r_{n}(d+1)}:=\Theta_{n}$. If $n$ is fixed, $\Theta_{n}$ is a bounded closed set and hence it is a compact set in $\mathbb{R}^{r_{n}(d+2)+1}$. Consider a map

[TABLE]

Note that $\mathcal{F}_{r_{n}}=H(\Theta_{n})$. Therefore, to show that $\mathcal{F}_{r_{n}}$ is a compact set, it suffices to show that $H$ is a continuous map due to the compactness of $\Theta_{n}$. Let $\boldsymbol{\theta}_{1,n},\boldsymbol{\theta}_{2,n}\in\Theta_{n}$, then

[TABLE]

Hence, for any $\epsilon>0$, we choose $\delta=\epsilon/\left(\frac{V_{n}}{4}(1\vee\|\boldsymbol{x}\|_{\infty})\sqrt{r_{n}(d+1)}\right)$,. When $\|\boldsymbol{\theta}_{1,n}-\boldsymbol{\theta}_{2,n}\|_{2}<\delta$, we have

[TABLE]

which implies that $H$ is a continuous map and hence $\mathcal{F}_{r_{n}}$ is a compact set for each fixed $n$. ∎

As a corollary of Lemma 2.1 and Theorem 2.1, we can easily obtain the existence of sieve estimator.

Corollary 2.1**.**

Based on the notations above, for each $n=1,2,\ldots$, there exists $\hat{f}_{n}:\Omega\to\mathcal{F}_{r_{n}}$, $\mathcal{A}/\mathcal{B}(\mathcal{F}_{r_{n}})$-measurable such that $\mathbb{Q}_{n}(\hat{f}_{n}(\omega))=\inf_{f\in\mathcal{F}_{r_{n}}}\mathbb{Q}_{n}(f)$.

3 Consistency

In this section, we are going to show the consistency of the neural network sieve estimator. The consistency result leans heavily on the following Uniform Law of Large Numbers. We start by considering a simple case with $V_{n}\equiv V$ for all $n$. In such a case, $\bigcup_{n}\mathcal{F}_{r_{n}}$ is not dense in $\mathcal{F}$ but rather in a subset of $\mathcal{F}$ with functions satisfying a certain smoothness condition.

Lemma 3.1**.**

Let $\epsilon_{1},\ldots,\epsilon_{n}$ be i.i.d. sub-Gaussian random variables with sub-Gaussian parameter $\sigma_{0}$. If $[r_{n}(d+2)+1]\log[r_{n}(d+2)+1]=o(n)$, we have

[TABLE]

Proof.

For any $\delta>0$, we have

[TABLE]

For (I), based on the Weak Law of Large Numbers, we know that there exists $N_{1}>0$ such that for all $n\geq N_{1}$ we have

[TABLE]

Now, we are going to evaluate (II). From the sub-Gaussianity of $\epsilon_{1},\ldots,\epsilon_{n}$, we know that $\epsilon_{i}(f(\boldsymbol{x}_{i})-f_{0}(\boldsymbol{x}_{i}))$ is also sub-Gaussian with mean 0 and sub-Gaussian parameter $\sigma_{0}|f(\boldsymbol{x}_{i})-f_{0}(\boldsymbol{x}_{i})|$. Hence, by using the Hoeffding inequality,

[TABLE]

From Proposition 2.2, we know that $\sup_{f\in\mathcal{F}_{r_{n}}}\|f\|_{n}\leq V$. Hence, based on Corollary 8.3 in van de Geer (2000), (II) will have an exponential bound if there exists some constant $C$, $\delta>0$ and $\sigma>0$ satisfying $V>\delta/\sigma$ and

[TABLE]

Now, we are going to show that (3.1) holds in our case. It follows from Theorem 14.5 in Anthony and Bartlett (2009), which gives an upper bound of the covering number for $\mathcal{F}_{r_{n}}$,

[TABLE]

where $\tilde{A}_{r_{n},d,V}=\left(e[r_{n}(d+2)+1]V^{2}/(V-4)\right)^{r_{n}(d+2)+1}$. By letting

[TABLE]

and noting that $V^{2}-eV+4e\geq 0$ for all $V$, we have $\log\frac{[r_{n}(d+2)+1]V^{2}}{V-4}\geq\log\frac{V^{2}}{V-4}\geq\log\frac{e(V-4)}{V-4}=1$. Then,

[TABLE]

Note that

[TABLE]

we have $H(\epsilon,\mathcal{F}_{r_{n}},\|\cdot\|_{n})\leq H(\epsilon,\mathcal{F}_{r_{n}},\|\cdot\|_{\infty})$. Then

[TABLE]

Clearly, $2\sqrt{2}A_{r_{n},d,V}^{1/2}V\geq V$. Under the assumption of $[r_{n}(d+2)+1]\log[r_{n}(d+2)+1]=o(n)$, for any $\delta>0$, there exists $N_{2}>0$ such that for all $n\geq N_{2}$,

[TABLE]

i.e. (3.1) holds with $C=1$ and $n\geq N_{2}$. Hence, based on Corollary 8.3 in van de Geer (2000), for $n\geq N_{2}$,

[TABLE]

Since $\int_{0}^{V}H^{1/2}(\epsilon,\mathcal{F}_{r_{n}},\|\cdot\|_{n})\textrm{d}\epsilon<\infty$, we can take $\sigma\to\infty$ in (3.2) to get

[TABLE]

Let $N_{3}=\frac{64V^{2}}{\delta^{2}}\log\frac{2}{\delta}$, then for $n\geq\max\{N_{2},N_{3}\}$, we have

[TABLE]

Thus, we conclude that for any $\delta>0$, by taking $n\geq\max\{N_{1},N_{2},N_{3}\}$, we have

[TABLE]

which proves the desired result. ∎

Remark 3.1**.**

Lemma 3.1 shows that if we have a fixed number of features, the desired Uniform Law of Large Numbers holds when the number of hidden units in the neural network sieve does not grow too fast.

Now, we are going to extend the result to a more general case. In Lemma 3.1, we assume that the errors $\epsilon_{1},\ldots,\epsilon_{n}$ are i.i.d. sub-Gaussian and $V_{n}\equiv V$. In the following lemma, we are going to relax both restrictions.

Lemma 3.2**.**

Under the assumption of

[TABLE]

we have

[TABLE]

Proof.

As in the proof of Lemma 3.1, it suffices to show that

[TABLE]

By using the Markov’s inequality, (3.3) holds if we can show

[TABLE]

Note that $\mathbb{E}[\epsilon]=0$ and each $f\in\mathcal{F}_{r_{n}}$ has its corresponding parametrization $\boldsymbol{\theta}_{n}$. Since $\boldsymbol{\theta}_{n}$ is in a compact set, there exists a sequence $\boldsymbol{\theta}_{n,k}\to\boldsymbol{\theta}_{n}$ as $k\to\infty$ with $\boldsymbol{\theta}_{n,k}\in\mathbb{Q}^{r_{n}(d+2)+1}\cap([-V_{n},V_{n}]^{r_{n}+1}\times[-M_{n},M_{n}]^{r_{n}(d+1)})$. Each $\boldsymbol{\theta}_{n,k}$ corresponds to a function $f_{k}\in\mathcal{F}_{r_{n}}$. Based on continuity, we have $f_{k}(\boldsymbol{x})\to f(\boldsymbol{x})$ for each $\boldsymbol{x}\in\mathcal{X}$. From Example 2.3.4 in van der Vaart and Wellner (1996), we know that $\mathcal{F}_{r_{n}}$ is $P$-measurable. Based on symmetrization inequality, we have

[TABLE]

where $\xi_{1},\ldots,\xi_{n}$ are i.i.d. Rademacher random variables independent of $\epsilon_{1},\ldots,\epsilon_{n}$. Based on the Strong Law of Large Numbers, there exists $N_{1}>0$, such that for all $n\geq N_{1}$,

[TABLE]

For fixed $\epsilon_{1},\ldots,\epsilon_{n}$, $\sum_{i=1}^{n}\xi_{i}\epsilon_{i}(f(\boldsymbol{x}_{i})-f_{0}(\boldsymbol{x}_{i}))$ is a sub-Gaussian process indexed by $f\in\mathcal{F}_{r_{n}}$. Suppose that $(\Xi,\mathcal{C},\mu)$ is the probability space on which $\xi_{1},\ldots,\xi_{n}$ are defined and let $Y(f,\omega)=\sum_{i=1}^{n}\xi_{i}(\omega)\epsilon_{i}(f(\boldsymbol{x}_{i})-f_{0}(\boldsymbol{x}_{i}))$ with $f\in\mathcal{F}_{r_{n}}$ and $\omega\in\Xi$. As we have shown above, we have $f_{k}\to f$ and by continuity, $Y(f_{k},\omega)\to Y(f,\omega)$ for any $\omega\in\Xi$. This shows that $\{Y(f,\omega),f\in\mathcal{F}_{r_{n}}\}$ is a separable sub-Gaussian process. Hence Corollary 2.2.8 in van der Vaart and Wellner (1996) implies that there exists a universal constant $K$ and for any $f_{n}^{*}\in\mathcal{F}_{r_{n}}$ with $n\geq N_{1}$,

[TABLE]

where the second equality follows from Proposition 2.2. For $f,g\in\mathcal{F}_{r_{n}}$,

[TABLE]

so that the last inequality follows by noting that

[TABLE]

We then evaluate these two terms. For the first term, for $n\geq N_{1}$, by Cauchy-Schwarz inequality, we have

[TABLE]

By choosing $f_{n}^{*}=\pi_{r_{n}}f_{0}$ and using the universal approximation theorem introduced by Hornik, Stinchcombe and White (1989), we know that $\sup_{x\in\mathcal{X}}|f_{n}^{*}(\boldsymbol{x}_{i})-f_{0}(\boldsymbol{x}_{i})|\to 0$ as $n\to\infty$. Therefore, for any $\zeta>0$, there exists $N_{2}>0$, such that for all $n\geq N_{2}$,

[TABLE]

By choosing $n\geq N_{1}\vee N_{2}$, we get

[TABLE]

For the second term, we use the same bound from Theorem 14.5 in Anthony and Bartlett (2009) as we did in the proof of Lemma 3.1:

[TABLE]

where $\tilde{B}_{r_{n},d,V_{n}}=\left(2\sqrt{\sigma^{2}+1}e[r_{n}(d+2)+1]V_{n}^{2}/(V_{n}-4)\right)^{r_{n}(d+2)+1}$. Let

[TABLE]

where $N_{3}$ is chosen to satisfy $r_{n}(d+2)+1\geq 2\sqrt{\sigma^{2}+1}$. The last inequality then follows by noting that $V_{n}^{2}-V_{n}+4\geq 0$ for all $V_{n}$ so that $\log\frac{[r_{n}(d+1)+1]V_{n}^{2}}{V_{n}-4}\geq\log\frac{2\sqrt{\sigma^{2}+1}(V_{n}-4)}{V_{n}-4}=\log(2\sqrt{\sigma^{2}+1})$. We also have

[TABLE]

and hence for all $n\geq N_{1}\vee N_{3}$,

[TABLE]

which implies that

[TABLE]

where the last part follows by noting that $\log\frac{V_{n}^{2}}{V_{n}-4}\sim\log V_{n}$. Under the assumption given in the Lemma, there exists $N_{4}>0$, such that for all $n\geq N_{4}$, we have

[TABLE]

Therefore, by choosing $n\geq N_{1}\vee N_{2}\vee N_{3}\vee N_{4}$, we get

[TABLE]

i.e. $\mathbb{E}_{\xi}\left[\sup_{f\in\mathcal{F}_{r_{n}}}\left|\frac{1}{n}\sum_{i=1}^{n}\xi_{i}\epsilon_{i}(f(\boldsymbol{x}_{i})-f_{0}(\boldsymbol{x}_{i}))\right|\right]\to 0$ a.s.. Moreover, based on what we have shown, for a sufficiently large $n$, we have

[TABLE]

Since $\mathbb{E}_{\epsilon}\left[\sqrt{\sigma^{2}+1}\|\pi_{r_{n}}f_{0}-f_{0}\|_{\infty}+4\sqrt{2}KB_{r_{n},d,V_{n}}^{1/2}n^{-1/2}V_{n}\right]=\sqrt{\sigma^{2}+1}\|\pi_{r_{n}}f_{0}-f_{0}\|_{\infty}+4\sqrt{2}KB_{r_{n},d,V_{n}}^{1/2}n^{-1/2}V_{n}\to 0<\infty$, by using the Generalized Dominated Convergence Theorem, we know that

[TABLE]

which completes the proof. ∎

Based on the above lemmas, we are ready to state the theorem on the consistency of neural network sieve estimators.

Theorem 3.1**.**

Under the notation given above, if

[TABLE]

then

[TABLE]

Proof.

Since $Q$ is continuous at $f_{0}\in\mathcal{F}$ and $Q(f_{0})=\sigma^{2}<\infty$, for any $\epsilon>0$, we have

[TABLE]

Hence, based on Lemma 2.1, Lemma 3.2 and Corollary 2.6 in White and Wooldridge (1991), we have

[TABLE]

∎

Remark 3.2**.**

We discuss the condition (3.4) in Theorem 3.1 via some simple examples here. If $\alpha_{j}=\mathcal{O}(1)$ for $j=1,\ldots,r_{n}$, then $V_{n}=\mathcal{O}(r_{n})$ and

[TABLE]

Therefore, a possible growth rate for the number of hidden units in a neural network is $r_{n}=o\left((n/\log n)^{1/3}\right)$. On the other hand, if we have a slow growth rate for the number of hidden units in the neural network, such as $r_{n}=\log V_{n}$, then we have

[TABLE]

Hence, a possible growth rate for the upper bound of the weights from the hidden layer to the output layer is $V_{n}=o\left(n^{1/2}/\log n\right)$.

4 Rate of Convergence

To obtain the rate of convergence for neural network sieves, we apply Theorem 3.4.1 in van der Vaart and Wellner (1996)van der Vaart and Wellner (1996).

Lemma 4.1**.**

Let $f_{n}^{*}=\pi_{r_{n}}f_{0}\in\mathcal{F}_{r_{n}}$. Given the above notations, for every $n$ and $\delta>8\|f_{n}^{*}-f_{0}\|_{n}$, we have

[TABLE]

Proof.

Note that

[TABLE]

In order to show the result, we first provide an upper bound for $Q_{n}(f_{n}^{*})-Q_{n}(f)$ in terms of $\|f-f_{n}^{*}\|_{n}$. Due to the fact that $\|\cdot\|_{n}$ is a pseudo-norm, the triangle inequality gives

[TABLE]

Therefore, we have

[TABLE]

so that for every $f$ satisfying $\|f-f_{n}^{*}\|_{n}^{2}\geq 16\|f_{n}^{*}-f_{0}\|_{n}^{2}$, i.e., $\|f-f_{n}^{*}\|_{n}\geq 4\|f_{n}^{*}-f_{0}\|_{n}$, we have

[TABLE]

By squaring both sides, we obtain

[TABLE]

and hence

[TABLE]

∎

Lemma 4.2**.**

For every sufficiently large $n$ and $\delta>8\|f_{n}^{*}-f_{0}\|_{n}$, we have

[TABLE]

Proof.

Note that

[TABLE]

we have

[TABLE]

Hence, by using the similar arguments as in the proof of Lemma 3.2 and applying Corollary 2.2.8 in van der Vaart and Wellner (1996)van der Vaart and Wellner (1996), we have

[TABLE]

where the last inequality follows since $f_{n}^{*}\in\mathcal{F}_{r_{n}}$ for a large enough $n$. ∎

Now we are ready to apply Theorem 3.4.1 in van der Vaart and Wellner (1996) to obtain the rate of convergence for neural network sieve estimators.

Theorem 4.1**.**

Based on the above notations, if

[TABLE]

then

[TABLE]

Proof.

Use the same bound from Theorem 14.5 in Anthony and Bartlett (2009), we have

[TABLE]

where $\tilde{C}_{r_{n},d,V_{n}}=\frac{e[r_{n}(d+2)+1]V_{n}^{2}}{V_{n}-4}>e$. Then from Lemma 3.8 in Mendelson (2003), for $\delta<1$,

[TABLE]

Define $h:\delta\mapsto\phi_{n}(\delta)/\delta^{\alpha}=[r_{n}(d+2)+1]^{1/2}\delta^{1-\alpha}\sqrt{\log\frac{\tilde{C}_{r_{n},d,V_{n}}}{\delta}}$. Since for $0<\delta<1$ and $1<\alpha<2$

[TABLE]

$\delta\mapsto\phi_{n}(\delta)/\delta^{\alpha}$ is decreasing on $(0,\infty)$. Let $\rho_{n}\lesssim\|\pi_{r_{n}}f_{0}-f_{0}\|_{n}^{-1}$. Note that

[TABLE]

and

[TABLE]

we have

[TABLE]

Therefore, for

[TABLE]

we have $\rho_{n}^{2}\phi_{n}\left(\frac{1}{\rho_{n}}\right)\lesssim\sqrt{n}$. Based on these observation, Lemma 4.1, Lemma 4.2 and Theorem 3.4.1 in van der Vaart and Wellner (1996)van der Vaart and Wellner (1996) imply that

[TABLE]

By using the triangle inequality, we can further get

[TABLE]

∎

Remark 4.1**.**

Recall that a sufficient condition to ensure consistency is $r_{n}(d+2)V_{n}^{2}\log[r_{n}V_{n}(d+2)]=o(n)$. Under such a condition, $r_{n}(d+2)\log[r_{n}V_{n}(d+2)]\leq n$, the rate of convergence can be simplified to

[TABLE]

If we assume $f_{0}\in\mathcal{F}$ where $\mathcal{F}$ is the space of functions with finite first absolute moments of the Fourier magnitude distributions, i.e.,

[TABLE]

where $\mu_{f}$ is a complex measure on $\mathbb{R}^{d}$. $|\mu_{f}|$ denotes the total variation of $\mu_{f}$, i.e., $|\mu|(A)=\sup\sum_{n=1}^{\infty}|\mu(A_{n})|$ and the supremum is taken over all measurable partitions $\{A_{n}\}_{n=1}^{\infty}$ of $A$. $\|\boldsymbol{a}\|_{1}=\sum_{i=1}^{d}|a_{i}|$ for $\boldsymbol{a}=[a_{1},\ldots,a_{d}]^{T}\in\mathbb{R}^{d}$. Theorem 3 in Makovoz (1996) shows that $\delta_{n}:=\|f_{0}-\pi_{r_{n}}f_{0}\|_{n}\lesssim r_{n}^{-1/2-1/(2d)}$. Therefore, if we let $d$ fixed and $\rho_{n}=\delta_{n}^{-1}$ and $V_{n}\equiv V$ in the proof of Theorem 4.1, $\delta_{n}$ must also satisfy the following inequality:

[TABLE]

One possible choice of $r_{n}$ to satisfy such condition is $r_{n}\asymp(n/\log n)^{\frac{d}{2+d}}$. In such a case, we obtain

[TABLE]

which is the same rate obtained in Chen and Shen (1998). It is interesting to note that in the case where $d=1$, we have $\|\hat{f}_{n}-f_{0}\|_{n}=\mathcal{O}_{p}\left((n/\log n)^{-1/3}\right)$. Such rate is close to the $\mathcal{O}_{p}(n^{-1/3})$, which is the convergence rate in non-parametric least square problems when the class of functions considered has bounded variation in $\mathbb{R}$ (see Example 9.3.3 in van de Geer (2000)). As shown in Proposition 7.3 in the Appendix, $\mathcal{F}_{r_{n}}$ is a class of functions with bounded variation in $\mathbb{R}$. Therfore, the convergence rate we obtained makes sense.

5 Asymptotic Normality

To establish the asymptotic normality of sieve estimator for neural network, we follow the idea in Shen (1997) and start by calculating the Gâteaux derivative of the empirical criterion function $\mathbb{Q}_{n}(f)=n^{-1}\sum_{i=1}^{n}(y_{i}-f(\boldsymbol{x}_{i}))^{2}$,

[TABLE]

Then the remainder of first-order functional Taylor series expansion is

[TABLE]

As will be seen in the proof of asymptotic normality, the rate of convergence for the empirical process $\{n^{-1/2}\sum_{i=1}^{n}\epsilon_{i}(f(\boldsymbol{x}_{i})-f_{0}(\boldsymbol{x}_{i})):f\in\mathcal{F}_{r_{n}}\}$ plays an important role. Here we establish a lemma, which will be used to find the desired rate of convergence.

Lemma 5.1**.**

Let $X_{1},\ldots,X_{n}$ be independent random variables with $X_{i}\sim P_{i}$. Define the empirical process $\{\nu_{n}(f)\}$ as

[TABLE]

Let $\mathcal{F}_{n}=\{f:\|f\|_{\infty}\leq V_{n}\}$, $\epsilon>0$ and $\alpha\geq\sup_{f\in\mathcal{F}_{n}}n^{-1}\sum_{i=1}^{n}\textrm{Var}[f(X_{i})]$ be arbitrary. Define $t_{0}$ by $H(t_{0},\mathcal{F}_{n},\|\cdot\|_{\infty})=\frac{\epsilon}{4}\psi(M,n,\alpha)$, where $\psi(M,n,\alpha)=M^{2}/\left[2\alpha\left(1+\frac{MV_{n}}{2\sqrt{n}\alpha}\right)\right]$. If

[TABLE]

for some $0<r<2$ and $u\in(0,a]$, where $a$ is a small positive number, and there exists a positive constant $K_{i}=K_{i}(r,\epsilon)$, $i=1,2$ such that

[TABLE]

we have

[TABLE]

Proof.

The proof of the lemma is similar to the proof of Corollary 2.2 in Alexander (1984) and the proof of Lemma 1 in Shen and Wong (1994). Since $H(u,\mathcal{F}_{n},\|\cdot\|_{\infty})\leq A_{n}u^{-r}$ for some $0<r<2$, we have

[TABLE]

Based on the assumption of

[TABLE]

we have $t_{0}\leq\left[\frac{4A_{n}}{\epsilon\psi}\right]^{1/r}$. Note that $\psi(M,n,\alpha)\geq M^{2}/(4\alpha)$ if $M\leq 3\sqrt{n}\alpha/V_{n}$ and $2(\sqrt{n}\alpha+MV_{n}/3)\leq 4MV_{n}/3$ if $M\geq 3\sqrt{n}\alpha/V_{n}$ and hence $\psi(M,n,\alpha)\geq 3\sqrt{n}M/(4V_{n})$. In summary,

[TABLE]

Therefore, if $M\geq 3\sqrt{n}\alpha/V_{n}$,

[TABLE]

where $\tilde{K}_{1}=2^{9}\epsilon^{-3/2}(2-r)^{-1}\left(\frac{4}{\epsilon}\right)^{\frac{1}{r}-\frac{1}{2}}\left(\frac{3}{4}\right)^{\frac{1}{2}-\frac{1}{r}}$. Hence

[TABLE]

where $K_{1}=\tilde{K}_{1}^{\frac{2r}{r+2}}$. On the other hand, if $M<3\sqrt{n}\alpha/V_{n}$,

[TABLE]

where $\tilde{K}_{2}=2^{9}\epsilon^{-3/2}(2-r)^{-1}\left(\frac{4}{\epsilon}\right)^{\frac{1}{r}-\frac{1}{2}}\left(\frac{1}{4}\right)^{\frac{1}{2}-\frac{1}{r}}$. Hence

[TABLE]

where $K_{2}=\tilde{K}_{2}^{r/2}$. In conclusion, if $M\geq K_{1}A_{n}^{\frac{2}{r+2}}V_{n}^{\frac{2-r}{r+2}}n^{\frac{r-2}{2(r+2)}}\vee K_{2}A_{n}^{1/2}\alpha^{\frac{2-r}{4}}$, then $2^{8}\epsilon^{-3/2}I\left(\frac{\epsilon M}{64\sqrt{n}},t_{0}\right)<M$. By Theorem 2.1 in Alexander (1984), we have the desired result. ∎

As a Corollary to Lemma 5.1, we can show that the supremum of the empirical process $\{n^{-1/2}\sum_{i=1}^{n}\epsilon_{i}(f(\boldsymbol{x}_{i})-f_{0}(\boldsymbol{x}_{i})):f\in\mathcal{F}_{r_{n}}\}$ converges to 0 in probability.

Corollary 5.1**.**

Let $\rho_{n}$ satisfy $\rho_{n}\|\hat{f}_{n}-f_{0}\|_{n}=\mathcal{O}_{p}(1)$ and $\mathcal{F}_{r_{n}}$ be the class of neural network sieves as defined in (1.2). Suppose that $\mathbb{E}[|\epsilon|^{2+\lambda}]<\infty$ for some $\lambda>0$. Then under the conditions

• (C1)

$r_{n}(d+2)V_{n}\log[r_{n}V_{n}(d+2)]=o(n^{1/4})$;

• (C2)

$n\rho_{n}^{-2}/V_{n}^{\lambda}=o(1)$,

we have

[TABLE]

Proof.

To establish the desired result, we apply the truncation device.

[TABLE]

For (I), we can apply Lemma 5.1 directly. Note that $|\epsilon\mathbb{I}_{\{|\epsilon|\leq V_{n}\}}(f-f_{0})(\boldsymbol{x})|\leq V_{n}(V_{n}+\|f_{0}\|_{\infty})\lesssim V_{n}^{2}$ since $\|f_{0}\|_{\infty}<\infty$ and for $0<\eta<1$,

[TABLE]

where $\tilde{C}_{r_{n},d,V_{n}}=\frac{e[r_{n}(d+2)+1]V_{n}^{2}}{V_{n}-4}$ and

[TABLE]

Therefore, equation (5.1) is satisfied with $r=1$ and $A_{n}=2C_{r_{n},d,V_{n}}$. Following from Lemma 5.1, for $M\gtrsim C_{r_{n},d,V_{n}}^{2/3}V_{n}^{2/3}n^{-1/6}\vee C_{r_{n},d,V_{n}}^{1/2}\alpha^{1/4}$, we have $(I)\leq 5\exp\left\{-(1-\epsilon)\psi(M,n,\alpha)\right\}$ and hence

[TABLE]

Since by using (C1),

[TABLE]

For (II), by using the Cauchy-Schwarz inequality, we have

[TABLE]

Then it follows from the Markov inequality that

[TABLE]

Based on condition (C2), we have $(II)\to 0$, and

[TABLE]

Combining the results we obtained above, we get

[TABLE]

∎

Remark 5.1**.**

Condition (C2) can be further simplified using the results from Theorem 4.1. If

[TABLE]

then

[TABLE]

It follows from condition (C1) that

[TABLE]

For simplicity, we assume that $\rho_{n}^{-1}\asymp\sqrt{r_{n}(d+2)\log n/n}$, which holds for functions having finite first absolute moments of the Fourier magnitude distributions as discussed at the end of section 4.4. Then in this case,

[TABLE]

so that condition (C2) becomes $r_{n}(d+2)\log n/V_{n}^{\lambda}\to 0$.

Now we are going to establish the asymptotic normality for neural network estimators. For $f\in\{f\in\mathcal{F}_{r_{n}}:\|f-f_{0}\|_{n}\leq\rho_{n}^{-1}\}$, we consider a local alternative

[TABLE]

where $0\leq\delta_{n}=\eta_{n}^{1/2}=o(n^{-1/2})$ is chosen such that $\rho_{n}\delta_{n}=o(1)$ and $\iota(\boldsymbol{x})\equiv 1$.

Theorem 5.1** (Asymptotic Normality).**

Suppose that $0\leq\eta_{n}=o(n^{-1})$ and conditions (C1) and (C2) in Corollary 5.1 hold. We further assume that the following two conditions hold

• (C3)

$\sup_{f\in\mathcal{F}_{r_{n}}:\|f-f_{0}\|_{n}\leq\rho_{n}^{-1}}\|\pi_{r_{n}}\tilde{f}_{n}(f)-\tilde{f}_{n}(f)\|_{n}=\mathcal{O}_{p}(\rho_{n}\delta_{n}^{2})$;

• (C4)

$\sup_{f\in\mathcal{F}_{r_{n}}:\|f-f_{0}\|_{n}\leq\rho_{n}^{-1}}\frac{1}{n}\sum_{i=1}^{n}\epsilon_{i}\left(\pi_{r_{n}}\tilde{f}_{n}(f)(\boldsymbol{x}_{i})-\tilde{f}_{n}(f)(\boldsymbol{x}_{i})\right)=\mathcal{O}_{p}(\delta_{n}^{2})$,

then

[TABLE]

Before we proceed to the proof of the theorem, let us focus on the conditions given in the theorem. Note that if (C1) holds, we have

[TABLE]

so it is a sufficient condition to ensure the consistency of the neural network sieve estimator. As in Remark 3.2, we consider some simple scenarios here. If $V_{n}=\mathcal{O}(r_{n})$, then $r_{n}(d+2)V_{n}\log[r_{n}V_{n}(d+2)]=\mathcal{O}\left(r_{n}^{2}\log r_{n}\right)$ so that a possible growth rate for $r_{n}$ is $r_{n}=o\left(n^{1/8}/(\log n)^{2}\right)$. On the other hand, if $r_{n}=\log V_{n}$, then $r_{n}(d+2)V_{n}\log[r_{n}V_{n}(d+2)]=\mathcal{O}\left(V_{n}(\log V_{n})^{2}\right)$ and a possible growth rate for $V_{n}$ is $V_{n}=o(n^{1/4}/(\log n)^{2})$. Thus, in both cases, the growth rate required for the asymptotic normality of neural network sieve estimator is slower than the growth rate required for the consistency as given in Remark 3.2. One explanation is that due to the Universal Approximation Theorem, a neural network with one hidden layer can approximate a continuous function on compact support arbitrarily well if the number of hidden units is sufficiently large. Therefore, if the number of hidden units is too large, the neural network sieve estimator $\hat{f}_{n}$ may be very close to the best projector of the true function $f_{0}$ in $\mathcal{F}_{r_{n}}$ so that the error $\sum_{i=1}^{n}\left[\hat{f}_{n}(\boldsymbol{x}_{i})-f_{0}(\boldsymbol{x}_{i})\right]$ could be close to zero, resulting a small variation. By allowing slower growth rate of the number of hidden units can increase the variations of $\sum_{i=1}^{n}\left[\hat{f}_{n}(\boldsymbol{x}_{i})-f_{0}(\boldsymbol{x}_{i})\right]$, which makes the asymptotic normality more reasonable. On the other hand, condition (C3) and condition (C4) are similar conditions as in Shen (1997), which are known for conditions on approximation error. These conditions indicate that the approximation rate of a single layer neural network cannot be too slow, otherwise it may require a huge number of samples to reach the desired approximation error. Therefore, the conditions in the theorem can be considered as a trade-off between bias and variance.

Proof of Theorem 5.1.

The main idea of the proof is to use the functional Taylor series expansion for $\mathbb{Q}_{n}(f)$ and then carefully bound each term in the expansion. For any $f\in\{f\in\mathcal{F}_{r_{n}}:\|f-f_{0}\|_{n}\leq\rho_{n}^{-1}\}$,

[TABLE]

Note that

[TABLE]

and since $\delta_{n}=o(n^{-1/2})$, we can know that with probability tending to 1, $\|\tilde{f}_{n}(f)-f_{0}\|_{n}\leq\rho_{n}^{-1}$. Then replacing $f$ in (5.3) by $\hat{f}_{n}$ and $\pi_{r_{n}}\tilde{f}_{n}(f)$ as defined in (5.2), we get

[TABLE]

Subtracting these two equations yields

[TABLE]

Now note that

[TABLE]

where the last inequality follows from the Cauchy-Schwarz inequality. Since

[TABLE]

by the definition of $\hat{f}_{n}$, we have

[TABLE]

[TABLE]

where the last inequality follows by noting that $(1-\delta_{n})^{2}-1=-2\delta_{n}+\delta_{n}^{2}\leq\delta_{n}^{2}$. From the condition (C1), we can get

[TABLE]

Combining with Theorem 3.1, we obtain that $\|\hat{f}_{n}-f_{0}\|_{n}=o_{p}(1)$ and hence $\delta_{n}^{2}\|\hat{f}_{n}-f_{0}\|_{n}^{2}=o_{p}(\delta_{n}^{2})$. From condition (C3), we have

[TABLE]

Similarly, since $\rho_{n}\delta_{n}=o(1)$, we have

[TABLE]

Based on condition (C4), we know that

[TABLE]

and from Corollary 5.1, we also have

[TABLE]

It follows from these observations that

[TABLE]

which implies that

[TABLE]

By replacing $\iota$ with $-\iota$, we can obtain the same result and hence

[TABLE]

Therefore,

[TABLE]

and the desired result follows from the classical Central Limit Theorem. ∎

Theorem 5.1 can be used directly for hypothesis testing of neural network with one hidden layer if we know the variance of the random error $\sigma^{2}$. In practice, this is rarely the case. To perform hypothesis testing when $\sigma^{2}$ is unknown, it is natural to find a good estimator of $\sigma^{2}$ and use a “plug-in” test statistic. A natural estimator for $\sigma^{2}$ is

[TABLE]

We then need to establish the asymptotic normality for the statistic $\frac{1}{\hat{\sigma}_{n}\sqrt{n}}\sum_{i=1}^{n}\left[\hat{f}_{n}(\boldsymbol{x}_{i})-f_{0}(\boldsymbol{x}_{i})\right]$.

Theorem 5.2** (Asymptotic Normality for Plug-in Statistic).**

Suppose that $f_{0}\in C(\mathcal{X})$, where $\mathcal{X}\subset\mathbb{R}^{d}$ is a compact set and $0\leq\eta_{n}=o\left(n^{-1}\right)$. Then under the conditions as stated in Theorem 5.1, we have

[TABLE]

Proof.

Note that

[TABLE]

Based on the rate of convergence of $\hat{f}_{n}$ we obtained in Theorem 4.1 and condition (C1), we know that

[TABLE]

Under (C3), $\|\pi_{r_{n}}f_{0}-f_{0}\|_{n}^{2}=o\left(\rho_{n}^{2}\delta_{n}^{4}\right)=o(n^{-1/2})$ and under (C1), we have

[TABLE]

which implies that $\left\|\hat{f}_{n}-f_{0}\right\|_{n}^{2}=o_{p}(n^{-1/2})$. Moreover, by the same arguments as in the proof of Theorem 5.1, we can show that

[TABLE]

Therefore,

[TABLE]

Based on the Weak Law of Large Numbers, we know that $\frac{1}{n}\sum_{i=1}^{n}\epsilon_{i}^{2}=\sigma^{2}+o_{p}(1)$. Therefore,

[TABLE]

and it follows from the Slutsky’s Theorem and Theorem 5.1, we obtain

[TABLE]

∎

6 Simulation Studies

In this section, simulations were conducted to check the validity of the theoretical results obtained in the previous sections. We first used a simple simulation to show that it is hard for the parameter estimators in a neural network with one hidden layer to reach parametric consistency. Then the consistency of the neural network sieve estimators was examined under various simulation scenarios. Finally, we evaluated the asymptotic normality of the neural network sieve estimators.

6.1 Parameter Inconsistency

As mentioned in the introduction, due to the loss of identifiability of the parameters, the parameter estimators obtained from a neural network model are unlikely to be consistent. In this simulation, we use empirical results to confirm such observations. We simulated the response through the following model:

[TABLE]

where the total sample size $n=500$, $x_{1},\ldots,x_{n}\sim\textrm{i.i.d.}\mathcal{N}(0,1)$, $\epsilon_{1},\ldots,\epsilon_{n}\sim\textrm{i.i.d.}\mathcal{N}(0,0.1^{2})$. The true model is a single-layer neural network with two hidden units.

[TABLE]

When we conducted the simulation, we fitted the simulated data with a single-layer neural network to fit the data by setting the learning rate as 0.1 and performing 3e4 iterations for the back propagation. The cost after 3e4 iterations is 0.0106. Table 1 summarizes the estimated values of the parameters in this model.

[TABLE]

Based on the results in Table 1, it is clear that the estimators for most of the weights and biases (except $\alpha_{0}$) are far from reaching consistency. On the other hand, if we look at the curve of the true function and the curve of the fitted function as shown in Figure 1, we can see that most parts are fitted extremely well except for the tail parts. The approximation error $\|\hat{f}-f_{0}\|_{n}$ is almost zero as shown in the Figure. This suggests that we should study the asymptotic properties of the estimated function instead of the estimated parameters in the neural network.

6.2 Consistency for Neural Network Sieve Estimators

In this simulation, we are going to check the consistency result from Section 3 and the validity of the assumption made in Theorem 3.1. Based on our construction of the neural network sieve estimators, in each sieve space $\mathcal{F}_{r_{n}}$, there is a constraint on the $\ell_{1}$ norm for $\boldsymbol{\alpha}$: $\sum_{i=0}^{r_{n}}|\alpha_{i}|\leq V_{n}$. So finding the nearly optimal function in $\mathcal{F}_{r_{n}}$ for $\mathbb{Q}_{n}(f)$ is in fact a constrained optimization problem. A classical way to conduct this optimization is through introducing a Lagrange multiplier for each constraint. Nevertheless, it is usually hard to find an explicit connection between the Lagrange multiplier and the upper bound in the inequality constraint. Instead, we use the subgradient method as discussed in section 7 in Boyd and Mutapcic (2008). The basic idea is to update the parameter $\alpha_{0},\ldots,\alpha_{r_{n}}$ through

[TABLE]

where $\delta_{k}>0$ is a step size and $\delta_{k}$ is chosen to be $0.1/\log(e+k)$ throughout the simulation, which is known as a nonsummable diminishing step size rule. $g^{(k)}$ is a subgradient of the objective or the constraint function $\sum_{j=0}^{r_{n}}|\alpha_{j}|-V_{n}$ at $\boldsymbol{\alpha}^{(k)}$. More specifically, we take

[TABLE]

The updating equations of $\boldsymbol{\gamma}_{1},\ldots\boldsymbol{\gamma}_{r_{n}},\gamma_{0,1},\ldots,\gamma_{0,r_{n}}$ remains the same as those in the classical gradient descent algorithm.

We still used equation (6.1) to simulate response, but assumed that the random error $\epsilon_{1},\ldots,\epsilon_{n}$ are i.i.d. $\mathcal{N}(0,0.7^{2})$. For the true function $f_{0}(x)$, we considered the following three functions:

1. (1)

A neural network with a single hidden layer and 2 hidden units, which is the same as in equation (6.2). 2. (2)

A trigonometric function:

[TABLE] 3. (3)

A continuous function having a non-differential point

[TABLE]

We then trained a neural network using the subgradient method mentioned above and set the number of iterations used for fitting as 20,000. We chose the growth rate on the number of hidden units $r_{n}=n^{1/4}$ and the upper bound for $\ell_{1}$ norm of the weights and bias from the hidden layer to the output layer $V_{n}=10n^{1/4}$. Such choice satisfies the condition mentioned in Remark 3.2 and hence satisfies the condition in Theorem 3.1. We compared the errors $\|\hat{f}_{n}-f_{0}\|_{n}^{2}$ and the least square errors $\mathbb{Q}_{n}(\hat{f}_{n})$ under different sample sizes. The results are summarized in Table 2.

As we can see from Table 2, the errors $\|\hat{f}_{n}-f_{0}\|_{n}^{2}$ overall has a decreasing pattern as the sample size increases. There are some cases where the error becomes a little bit larger when the sample sizes increases (e.g. the errors using 500 samples in all scenarios is larger than those errors using 200 sample). One explanation is that the number of hidden units increase from 3 (for 200 samples) to 4 (for 500 samples) under our simulation setup, which adds variation to the estimation performance. Overall, the table shows that the estimated function $\hat{f}_{n}$ is indeed consistent in the sense that $\|\hat{f}_{n}-f_{0}\|_{n}=o_{p}^{*}(1)$. Figure 2 plots the fitted functions and the true function, from which we can straightforwardly visualize the result more and draw the conclusions.

6.3 Asymptotic Normality for Neural Network Sieve Estimators

The last part of the simulation focuses on the asymptotic normality derived in Theorem 5.1. We still considered the same types of true functions as described in section 6.2 but sampled the random errors from the standard normal distribution. In this simulation, we still used the subgradient method to obtain the fitted model. The number of iterations used for fitting was set at 20,000. What is different from section 6.2 is the growth rates for $r_{n}$ and $V_{n}$ set in this simulation. As mentioned in section 5, the growth rates required for asymptotic normality are slower than those required for consistency. Therefore, we chose $r_{n}=n^{1/8}$ and $V_{n}=10n^{1/10}$. Such choice satisfies the condition (C1) in Theorem 5.1. In order to get the normal Q-Q plot for $n^{-1/2}\sum_{i=1}^{n}\left[\hat{f}_{n}(\boldsymbol{x}_{i})-f_{0}(\boldsymbol{x}_{i})\right]$, we repeated the simulation 200 times.

Figure 3 to Figure 5 are the normal Q-Q plots under different nonlinear functions and various sample sizes. From the figures, we found that the statistic $n^{-1/2}\sum_{i=1}^{n}\left[\hat{f}_{n}(\boldsymbol{x}_{i})-f_{0}(\boldsymbol{x}_{i})\right]$ fit the normal distribution pretty well under all simulation scenarios. It is also worth to note that the Q-Q plots looks similar under all simulation scenarios. This is what we would expect since the limiting distribution for the statistic $n^{-1/2}\sum_{i=1}^{n}\left[\hat{f}_{n}(\boldsymbol{x}_{i})-f_{0}(\boldsymbol{x}_{i})\right]$ is $\mathcal{N}(0,1)$ under all scenarios. Another implication we can obtain from the Q-Q plots is that the statistic $n^{-1/2}\sum_{i=1}^{n}\left[\hat{f}_{n}(\boldsymbol{x}_{i})-f_{0}(\boldsymbol{x}_{i})\right]$ is robust to the choice of $f_{0}$. Therefore, as long as the true function $f_{0}$ is continuous, $\mathcal{N}(0,1)$ is a good asymptotic distribution for $n^{-1/2}\sum_{i=1}^{n}\left[\hat{f}_{n}(\boldsymbol{x}_{i})-f_{0}(\boldsymbol{x}_{i})\right]$, which facilitates hypothesis testing.

Besides the Q-Q plots, we also conducted the normality tests to check whether $n^{-1/2}$ $\sum_{i=1}^{n}\left[\hat{f}_{n}(\boldsymbol{x}_{i})-f_{0}(\boldsymbol{x}_{i})\right]$ follows the standard normal distribution. Specifically, we used the Shapiro-Wilks test and the Kolmogorov-Smirnov test to perform the normality test. Table 3 summarizes the $p$-values for both normality tests. As we observed from Table 3, in all cases, we failed to reject that $n^{-1/2}\sum_{i=1}^{n}\left[\hat{f}_{n}(\boldsymbol{x}_{i})-f_{0}(\boldsymbol{x}_{i})\right]$ follows the standard normal distribution.

7 Discussion

We have investigated the asymptotic properties, including the consistency, rate of convergence, and asymptotic normality for neural network sieve estimators with one hidden layer. While in practice, the number of hidden unites is often chosen ad hoc, it is important to note that the conditions in the theorems provide theoretical guidelines on choosing the number of hidden units for a neural network with one hidden layer to achieve the desired statistical properties. The validity of the conditions made in the theorems has also been checked through simulation results. Theorem 5.1 and Theorem 5.2 can be served as preliminary work for conducting hypothesis testing on $H_{0}:f_{0}=h_{0}$ for a fixed function $h_{0}$. However, there is currently no simple way to check conditions (C3) and (C4) in the theorem, which requires further researches on local entropy numbers for classes of neural networks.

The work conducted in this paper mainly focuses on sieve estimators based on neural networks with one hidden layer and standard sigmoid activation function. The work presented in this paper can be extended in several ways. The main theorems in this paper depend heavily on the covering number or the entropy number of the function class consisting of neural network with one hidden layer. Theorem 14.5 in Anthony and Bartlett (2009) provides a general upper bound for the covering number of a function class consisting of deep neural networks with Lipshitz continuous activation functions. Therefore, it is possible to extend our results discussed in this paper to a deep neural network with Lipshitz continuous activation functions. It is also worthwhile to investigate asymptotic properties of other commonly used deep learning models such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs).

When we train a deep neural network, we usually need to face an overfitting issue. In practice, regularization is frequently used to reduce overfitting. Another natural extension of the work discussed in this paper is to modify the loss function by involving some regularization terms. By taking regularization into account, we believe the theories could have a much broader application in real world scenarios.

Acknowledgment

X. Shen, C. Jiang, and Q. Lu are supported by the National Institute on Drug Abuse (Award No. K01DA033346 and No. R01DA043501). L. Sakhanenko is supported by NSF grant DMS- 1742881.

Appendix

In this appendix, we are going to explore some basic properties of the parameter space $(\mathcal{F},\|\cdot\|_{n})$ discussed in the main text.

Proposition 7.1**.**

The space $(\mathcal{F},\|\cdot\|_{n})$ is a pseudo-normed space.

Proof.

Note that $\|f\|_{n}=\left(\frac{1}{n}\sum_{i=1}^{n}f^{2}(\boldsymbol{x}_{i})\right)^{1/2}$.

1. (i)

Based on the definition of $\|\cdot\|_{n}$, it is clear that $\|f\|_{n}\geq 0$, for any $f\in\mathcal{F}$. 2. (ii)

For any $\lambda\in\mathbb{R}$ and $f\in\mathcal{F}$,

[TABLE] 3. (iii)

For any $f,g\in\mathcal{F}$,

[TABLE]

where we applied the triangle inequality to the classical Euclidean norm.

Therefore, we showed that $(\mathcal{F},\|\cdot\|_{n})$ is a pseudo-normed space. ∎

Proposition 7.2**.**

There is an pseudo-inner product on $\mathcal{F}$ such that $\|f\|^{2}=\langle f,f\rangle$ for any $f\in\mathcal{F}$. Moreover, the pseudo-inner product is given by

[TABLE]

Proof.

Based on the theorem attributed to Fréchet, von Neumann and Jordan (see for example, Proposition 14.1.2 in Blanchard and Brüning (2015)), to show the existence of the inner product, it suffices to check the parallelogram law of the pseudo-norm and the corresponding pseudo-inner product can be obtained via the polarization identity. To check to validity of the parallelogram law, we note that for any $f,g\in\mathcal{F}$,

[TABLE]

Hence, the parallelogram law is satisfied based on the pseudo-norm, and the pseudo-inner product does exist. By using the polarization identity, for any $f,g\in\mathcal{F}$, we have

[TABLE]

∎

Let

[TABLE]

be the class of functions of bounded variation in $\mathbb{R}$ (see Example 9.3.3 in van de Geer (2000)). The following proposition shows that $\mathcal{F}_{r_{n}}\subset\mathcal{G}$ for a fixed $n$.

Proposition 7.3**.**

For a fixed $n$, $\mathcal{F}_{r_{n}}\subset\mathcal{G}$.

Proof.

For any $f\in\mathcal{F}_{r_{n}}$, we have

[TABLE]

so that

[TABLE]

Without loss of generality, we assume that $\gamma_{j}\neq 0$ for $j=1,\ldots,r_{n}$. Note that

[TABLE]

where in the last inequality, we let $u_{j}=\gamma_{j}x+\gamma_{0,j}$. Clearly, $|\gamma_{j}|/\gamma_{j}=\textrm{sign}(\gamma_{j})$. Moreover, since

[TABLE]

for a fixed $n$, we have

[TABLE]

Therefore, $f\in\mathcal{G}$ and the desired result follows. ∎