On the minimax optimality and superiority of deep neural network learning over sparse parameter spaces

TL;DR

This paper demonstrates that deep neural networks are minimax optimal for certain sparse and discontinuous function classes, outperforming linear estimators and highlighting the benefits of parameter sharing.

Contribution

It provides a theoretical analysis showing deep learning's superiority over linear methods on sparse and discontinuous function classes, with insights into model complexity reduction.

Findings

Deep learning attains nearly minimax-optimal rates on sparse function classes.

Linear estimators are suboptimal on non-convex and sparse function classes.

Parameter sharing in neural networks reduces model complexity significantly.

Abstract

Deep learning has been applied to various tasks in the field of machine learning and has shown superiority to other common procedures such as kernel methods. To provide a better theoretical understanding of the reasons for its success, we discuss the performance of deep learning and other methods on a nonparametric regression problem with a Gaussian noise. Whereas existing theoretical studies of deep learning have been based mainly on mathematical theories of well-known function classes such as H\"{o}lder and Besov classes, we focus on function classes with discontinuity and sparsity, which are those naturally assumed in practice. To highlight the effectiveness of deep learning, we compare deep learning with a class of linear estimators representative of a class of shallow estimators. It is shown that the minimax risk of a linear estimator on the convex hull of a target function class does not differ from that of the original target function class. This results in the suboptimality of linear methods over a simple but non-convex function class, on which deep learning can attain nearly the minimax-optimal rate. In addition to this extreme case, we consider function classes with sparse wavelet coefficients. On these function classes, deep learning also attains the minimax rate up to log factors of the sample size, and linear methods are still suboptimal if the assumed sparsity is strong. We also point out that the parameter sharing of deep neural networks can remarkably reduce the complexity of the model in our setting.

Tables

Table 1. Table 1: Estimated error bound: deep learning vs. linear methods

Target class		${ℐ}_{\Phi }^0$
Deep learning	$\tilde{O}\left(n^{-\frac{2s}{2s+1}}\right)$	$\tilde{O}\left(n^{-1}\right)$ (Sec. 5.3)	$\tilde{O}\left(n^{-\frac{2\alpha }{2\alpha +1}}\right)\left(\alpha =\frac{1}{p}-\frac{1}{2}\right)$ (Sec. 5.4)
Reference	Suzuki (2019)	This work
Linear methods	$\Omega \left(n^{-\frac{2\gamma }{2\gamma +1}}\right)$ $\left(\gamma =s+\frac{1}{2}-\frac{1}{p}\right)$	$\Omega \left(n^{-\frac{1}{2}}\right)$ (Sec. 3.2, 5.3, 5.4)
Reference	Donoho and Johnstone (1998) Zhang et al. (2002)	This work

Table 2. Table 2: Correspondence between mathematical features and the real-world things

Feature	Real-world counterpart	Examples
nonparametric regression	real-world estimation problems	image denoising, classification problem, style transfer
basis function	localized pattern	pronunciation of vowels, painting style, fractal structure in nature
sparse combination of basis functions	real data	speech of one person, a painting, a noisy picture
sparsity	low dimensionality of actual information	natural language, handwriting alphabets

Equations

$$Y_i=f^{\circ }(X_i)+{\xi }_i,i=1,\dots ,n,$$

$$\underset{f^{\circ }\in {\mathcal{F}}^{\circ }}{\sup }\mathrm{E}\left[\Vert \hat{f}-f^{\circ }{\Vert }_{L^2}^2\right]$$

$$\underset{(X_i,Y_i)\mapsto \hat{f}}{\inf }\underset{f^{\circ }\in {\mathcal{F}}^{\circ }}{\sup }\mathrm{E}\left[\Vert \hat{f}-f^{\circ }{\Vert }_{L^2}^2\right],$$

$$\hat{f}(x)=\sum _{i=1}^n{Y}_i{\phi }_i(x;X_1,\dots ,X_n).$$

$$\underset{(X_i,Y_i)\mapsto \hat{f}}{\inf }\underset{f^{\circ }\in {\mathcal{F}}^{\circ }}{\sup }\mathrm{E}\left[\Vert \hat{f}-f^{\circ }{\Vert }_{L^2}^2\right]\le n^{-\gamma }\underset{\hat{f}:\text{linear}}{\inf }\underset{f^{\circ }\in {\mathcal{F}}^{\circ }}{\sup }\mathrm{E}\left[\Vert \hat{f}-f^{\circ }{\Vert }_{L^2}^2\right]$$

$$\sigma :\mathbb{R}\to \mathbb{R},\text{with}\underset{t\to \infty }{\lim }\sigma (t)=1,\underset{t\to -\infty }{\lim }\sigma (t)=0.$$

$$\text{minimize}\frac{1}{n}\sum _{i=1}^n(f(X_i)-Y_i{)}^2+\lambda (f)\text{subject to}f\in \mathcal{F},$$

$$\Vert v{\Vert }_{\infty }:=\underset{1\le i\le m}{\max }|v_i|,\Vert v{\Vert }_0:=\left|\{1\le i\le m\mid v_i\ne 0\}\right|$$

$$\Vert a{\Vert }_{{\ell }^p}:={\left(\sum _{i=1}^{\infty }|a_i{|}^p\right)}^{1/p},$$

$$Y_i=f^{\circ }(X_i)+{\xi }_i,i=1,2,\dots ,n.$$

$$\mathrm{E}[(f(X)-Y{)}^2]$$

$$=\mathrm{E}[(f(X)-f^{\circ }(X){)}^2]+{\sigma }^2$$

$$=\Vert f-f^{\circ }{\Vert }_{L^2}^2+{\sigma }^2.$$

$$R(\hat{f},f^{\circ }):=\mathrm{E}\left[\Vert \hat{f}-f^{\circ }{\Vert }_{L^2}^2\right].$$

$$\underset{\hat{f}}{\inf }\underset{f^{\circ }\in {\mathcal{F}}^{\circ }}{\sup }R(\hat{f},f^{\circ })=\underset{\hat{f}}{\inf }\underset{f^{\circ }\in {\mathcal{F}}^{\circ }}{\sup }\mathrm{E}\left[\Vert \hat{f}-f^{\circ }{\Vert }_{L^2}^2\right],$$

$$V(\epsilon )\le \frac{n{\epsilon }^2}{2{\sigma }^2},M(\delta )\ge \frac{2n{\epsilon }^2}{{\sigma }^2}+2\log 2.$$

$$\underset{\hat{f}}{\inf }\underset{f\in {\mathcal{F}}^{\circ }}{\sup }{\mathrm{P}}_f\left(\Vert \hat{f}-f{\Vert }_{L^2}\ge \frac{\delta }{2}\right)\ge \frac{1}{2},\underset{\hat{f}}{\inf }\underset{f\in {\mathcal{F}}^{\circ }}{\sup }{\mathrm{E}}_f\left[\Vert \hat{f}-f{\Vert }_{L^2}^2\right]\ge \frac{{\delta }^2}{8},$$

$$\displaystyle R(\widehat{f},f^{\circ})\leq 4\inf_{f\in\mathcal{F}}\|f-f^{\circ}\|_{L^{2}}^{2}+C\biggr{(}\frac{(F^{2}+\sigma^{2})V_{(\mathcal{F},\|\cdot\|_{L^{\infty}})}(\delta)}{n}+(F+\sigma)\delta\biggl{)}$$

$$\hat{f}(x)=\sum _{i=1}^n{Y}_i{\phi }_i(x;X^n),$$

$$\hat{f}(x)={\hat{f}}_L(x)+\phi (x;X^n),$$

$$\hat{f}(x):=(k(x,X_1),\dots ,k(x,X_n))(K+\lambda I_n{)}^{-1}(Y_1,\dots ,Y_n{)}^{\top },$$

$$\mathrm{conv}({\mathcal{F}}^{\circ }):=\left\{\sum _{i=1}^k{t}_i{f}_i|t_1,\dots ,t_k\ge 0,\sum _{i=1}^k{t}_i=1,f_1,\dots ,f_k\in {\mathcal{F}}^{\circ },k\ge 1\right\}.$$

$$\underset{\hat{f}:\mathrm{affine}}{\inf }\underset{f^{\circ }\in {\mathcal{F}}^{\circ }}{\sup }R(\hat{f},f^{\circ })=\underset{\hat{f}:\mathrm{affine}}{\inf }\underset{f^{\circ }\in \overline{\mathrm{conv}}({\mathcal{F}}^{\circ })}{\sup }R(\hat{f},f^{\circ }).$$

$$\underset{\hat{f}:\mathrm{affine}}{\inf }\underset{f^{\circ }\in {\mathcal{F}}^{\circ }}{\sup }R(\hat{f},f^{\circ })=\underset{\hat{f}:\mathrm{affine}}{\inf }\underset{f^{\circ }\in \mathrm{conv}({\mathcal{F}}^{\circ })}{\sup }R(\hat{f},f^{\circ }).$$

$$\hat{f}(x)=\phi (x;X^n)+\sum _{i=1}^n{Y}_i{\phi }_i(x;X^n).$$

$$R(\hat{f},h^{\circ })=\mathrm{E}\left[{\int }_{[0,1{]}^d}{\left(\hat{f}(x)-h^{\circ }(x)\right)}^2\mathrm{d}x\right]={\int }_{[0,1{]}^d}\mathrm{E}\left[{\left(\hat{f}(x)-h^{\circ }(x)\right)}^2\right]\mathrm{d}x$$

$${\left(\hat{f}(x)-h^{\circ }(x)\right)}^2$$

$$\displaystyle=\Biggl{(}\varphi(x;X^{n})+\sum_{i=1}^{n}\xi_{i}\varphi_{i}(x;X^{n})+\sum_{i=1}^{n}h^{\circ}(X_{i})\varphi_{i}(x;X^{n})-h^{\circ}(x)\Biggr{)}^{2}$$

$$\displaystyle\leq t\Biggl{(}\varphi(x;X^{n})+\sum_{i=1}^{n}\xi_{i}\varphi_{i}(x;X^{n})+\sum_{i=1}^{n}f^{\circ}(X_{i})\varphi_{i}(x;X^{n})-f^{\circ}(x)\Biggr{)}^{2}$$

$$\displaystyle\quad+(1-t)\Biggl{(}\varphi(x;X^{n})+\sum_{i=1}^{n}\xi_{i}\varphi_{i}(x;X^{n})+\sum_{i=1}^{n}g^{\circ}(X_{i})\varphi_{i}(x;X^{n})-g^{\circ}(x)\Biggr{)}^{2}$$

$$\displaystyle=t\left(\widehat{f}(x)\bigg{|}_{Y_{i}=f^{\circ}(X_{i})+\xi_{i}}-f^{\circ}(x)\right)^{2}+(1-t)\left(\widehat{f}(x)\bigg{|}_{Y_{i}=g^{\circ}(X_{i})+\xi_{i}}-g^{\circ}(x)\right)^{2}.$$

$$R(\hat{f},h^{\circ })\le tR(\hat{f},f^{\circ })+(1-t)R(\hat{f},g^{\circ }).$$

$$J_k(C):=\left\{a_0+\sum _{i=1}^k{a}_i{1}_{[t_i,1]}|t_i\in (0,1],|a_0|\le C,\sum _{i=1}^k|a_i|\le C\right\}$$

Full text

Keywords— neural network, deep learning, linear estimator, nonparametric regression, minimax optimality

1 Introduction

Deep learning has been successfully applied to a number of machine learning problems, including image analysis and speech recognition Schmidhuber (2015); Goodfellow et al. (2016). However, the rapid expansion of its applications has preceded a thorough theoretical understanding, and thus the theoretical properties of neural networks and their learning have not yet been fully understood. This paper aims to summarize recent developments in theoretical analyses of deep learning and to provide new approximation and estimation error bounds that theoretically confirm the superiority of deep learning to other representative methods.

In this section, we present an overview of the paper; here, we prioritize understandability over strict mathematical rigor. Some formal definitions and restrictions, such as for measurability and integrability, are presented in later sections.

1.1 Nonparametric regression

Throughout this paper, our intent is to demonstrate the superiority of the deep learning approach to other methods. To do so, we consider a simple nonparametric regression problem and compare the performance of various approaches in that setting. The nonparametric regression problem we analyze is formulated as follows:

We observe $n$ i.i.d. input–output pairs $(X_{i},Y_{i})\in[0,1]^{d}\times\mathbb{R}$ generated by the model

[TABLE]

where $\xi_{i}$ is an i.i.d. noise independent of inputs. The object is to estimate $f^{\circ}$ from the observed data.

This problem setting has been commonly used in statistical learning theory and is not limited to deep learning Yang and Barron (1999); Zhang et al. (2002); Tsybakov (2008). In this paper, we assume the noise follows a Gaussian distribution.

In this scenario, a neural network (architecture) is treated as a set of functions $\mathcal{F}\subset\{f:[0,1]^{d}\to\mathbb{R}\}$. Other estimation methods such as kernel ridge regression and wavelet threshold estimators are also regarded as such mappings Bishop (2006); Donoho and Johnstone (1994). In this paper, we evaluate the performance of estimators by the expected mean squared error $\mathrm{E}\left[\|\widehat{f}-f^{\circ}\|_{L^{2}}^{2}\right]$ (we call this quantity the “estimation error” for simplicity) dependent on $n$ following convention Wang et al. (2014); Schmidt-Hieber (2017); Suzuki (2019), where the expectation is taken with respect to the training data. Usually, the $L^{2}(\mathrm{P}_{X})$ norm (where $\mathrm{P}_{X}$ is the distribution of $X_{i}$) has been used in existing studies instead of the Lebesgue $L^{2}$ norm, but in evaluation the upper- (and lower-) boundedness of the density is typically assumed, so for simplicity, we treat $X_{i}$ as uniformly distributed.

If we fix the set of true functions $\mathcal{F}^{\circ}$ (called a hypothesis space),

[TABLE]

is the worst-case performance of the estimator $(X_{i},Y_{i})_{i=1}^{n}\mapsto\widehat{f}$. We are interested in its asymptotic convergence rate with respect to $n$, the sample size. The minimax rate is determined by the convergence rate of

[TABLE]

where $\inf$ is taken over all possible estimators. We compare this to the convergence rate of fixed (with respect to $n$) sequences of estimators determined by some learning procedure such as deep learning to evaluate how efficient the estimation method is.

As a competitor of deep learning, a class of “linear estimators” is considered. Here, we say an estimator is linear if it depends linearly on the outputs $Y_{i}$; it is expressed as

[TABLE]

This estimator class includes several practical estimators such as kernel ridge regression and the Nadaraya–Watson estimator. The minimax rate in the class of linear estimators can be slower under some settings; e.g.,

[TABLE]

holds for some $\gamma>0$ (for most cases, we consider only the polynomial order). Such situations were reported earlier by several authors Korostelev and Tsybakov (1993); Donoho and Johnstone (1998); Zhang et al. (2002). In terms of deep learning analysis, a comparison of deep learning with linear methods has been performed by Imaizumi and Fukumizu (2019). The present paper also shows the suboptimality of linear methods (Table 1) for sparse function classes, which we define later.

Our main contribution here is that we find a quite simple and natural function class $\mathcal{I}_{\Phi}^{0}$ for which deep learning attains nearly the optimal rate, whereas linear methods are not able to converge faster than the suboptimal rate $\mathrm{O}(n^{-1/2})$. In the next subsection, we explain how to treat and analyze deep learning in the context of statistical learning theory.

1.2 Related work on estimation of deep neural networks

Deep neural networks have a structure of alternating linear (or affine) transformations and nonlinear transformations; i.e., in one layer $x$ is transformed to $\rho(Wx-v)$, where $W$ is a matrix and $v$ is a vector and $\rho$ is a nonlinear function called an activation function. It is known that the repeated operation of this transformation gives a nice approximation of a wide class of nonlinear functions.

Traditionally, sigmoidal functions have been commonly used as activation functions:

[TABLE]

It is known that the set of functions realized by shallow networks with continuous sigmoidal activation is dense in any $L^{p}$ space unless the number of parameters is not limited Cybenko (1989). However, a similar result has also been shown for non-sigmoidal activation cases Sonoda and Murata (2017). In particular, the Rectified Linear Unit (ReLU) activation function $\rho(x)=\max\{x,0\}$ has shown practical performance Glorot et al. (2011) and is now widely used.

Basically, deep learning trains a network by minimizing the empirical risk with some regularization:

[TABLE]

where $\lambda(f)$ is the regularization term and $\mathcal{F}$ is the set of functions that are realizations of a specific neural network architecture. This optimization is usually carried out by Stochastic Gradient Descent (SGD) or a variant of it, and the output is not necessarily the global minimum Goodfellow et al. (2016). In the present paper, however, we do not treat this optimization aspect, and we assume an ideal optimization.

The number of parameters in deep learning tends to be much larger than the sample size, and hence without any regularization, deep models can overfit the training sample. To overcome this issue, existing studies have utilized sparse regularization to obtain networks with a small number of nonzero parameters. This enables us to obtain a tight estimate of the error bounds using the result of approximation error analysis (see Section 2.2).

Yarotsky (2017) reported the effectiveness of ReLU activation in terms of approximation ability, and the result has been exploited in estimation theory for deep learning Schmidt-Hieber (2017); Suzuki (2019). Their target function classes are Hölder space $C^{s}$ and Besov space $B_{p,q}^{s}$, which are compatible with functional analysis or the theory of differential equations. Schmidt-Hieber (2017) also pointed out that deep learning is superior to linear estimators when the target function is of the form $f^{\circ}(x)=g(w^{\top}x)$, with $g$ having some Hölder smoothness. In addition, Imaizumi and Fukumizu (2019) treated estimation theory for piecewise smooth functions using the approximation theory described in Petersen and Voigtlaender (2018). This paper investigates new target classes to demonstrate the effectiveness of deep learning with ReLU activation.

1.3 Contribution of this paper

The situations where deep learning shows speriority to linear estimators have been studied with particular theories:

• •

$f^{\circ}(x)=g(w^{\top}x)$ with $g$ being smooth Schmidt-Hieber (2017),

• •

piece-wise smooth functions Imaizumi and Fukumizu (2019),

• •

Besov spaces in a certain rage of parameters Suzuki (2019).

These situations can be understood mathematically through the non-convexity of models. To highlight this property, we introduce a sparse function class (and therefore non-convex) and investigate the generalization ability of deep learning and linear estimators over the space to see how sparsity (non-convexity) affects the estimation error. We indeed show that linear estimators have a critical disadvantage when the target class is non-convex (Section 3). In contrast, deep learning is shown to have the optimality over the sparse function classes we define (Section 5). These classes are natural in the sense we describe in the next paragraph.

The major difference between existing studies and this work is that we assume an explicit sparsity of target classes, which are defined parametrically. This kind of scenario seems to occur in practice; for example, speech data for a specific person are supposed to be a sparse linear combination of the person’s pronunciation of each letter, and paintings by a specific painter may be regarded as combinations of patterns (see Table 2). Indeed, when we carry out the wavelet expansion to a natural image, its relatively few large coefficients can well reconstruct the original image, as is exploited in the field of compressive sensing Candès and Wakin (2008).

We reflect this property to define new function classes (Section 4). In particular, we introduce a function class $\mathcal{I}_{\Phi}^{0}$ and $\mathcal{K}_{\Psi}^{p}$ (and $\mathcal{J}_{\psi}^{p}$ as well) with a parameter $p>0$ controlling the “sparsity” of the function class. Figure 1 gives an interpretation of the motivations and definitions of these function classes. More precisely, $p$ controls the sparsity of coefficients of “infinite sparse combination” appearing in Figure 1 through a criterion called “weak $\ell^{p}$ norm” (see Definition 4.4 for details). Here, “combination” roughly means linear combination, but precisely includes affine transform in the input; i.e., the combination of $f_{1},\ldots,f_{m}$ can be generally expressed as $f=\sum_{i=1}^{m}c_{i}f_{i}(A_{i}\cdot-b_{i})$.

On the basis of our new function classes, we show that deep learning is superior to linear methods and actually attains nearly the minimax-optimal rate over each target class (Table 1). As an extreme case (corresponding to the sparsity level $p=0$), we treat the class of piecewise constant functions, for which the convergence rate of the linear estimators is $\Omega(n^{-1/2})$, whereas deep learning attains the near-minimax rate $\widetilde{\mathrm{O}}(n^{-1})$. This quite simply demonstrates the scenario described in Imaizumi and Fukumizu (2019). For $0<p<1$, we also show that deep learning attains the nearly minimax-optimal rate $\widetilde{\mathrm{O}}(n^{-\frac{2\alpha}{2\alpha+1}})$ in estimating a function in $\mathcal{K}^{p}_{\Psi}$, where $\alpha=1/p-1/2$. Here, we have $\frac{2\alpha}{2\alpha+1}>\frac{1}{2}$, and the difference between deep and linear becomes larger as $p$ becomes smaller (i.e., as the sparsity becomes stronger). $K_{\Psi}^{p}$ has another parameter, $\beta$, which controls the rate of decay of coefficients of the function class. Surprisingly, we even find that the minimax rate of linear estimators can become arbitrarily slow under the same sparsity $p$ (and the same order of covering entropy) if the value of $\beta$ is varied. Although we do not yet have the upper bound for the convergence rate of deep learning over the range of parameter values producing this situation (see Theorem 4.15 and Remark 4.16), this indicates that the difference between deep learning and linear estimators could be arbitrary large. These differences essentially arise from the non-convexity of the model. That is, as the non-convexity of the model becomes stronger, the difference becomes larger.

In addition, we see that deep learning takes advantage of wavelet expansions with sparsity because a neural network can efficiently approximate functions of the form $\sum_{i}c_{i}f(A_{i}\cdot-b_{i})$ if its subnetwork can approximate the “basis” function $f$ precisely as is also mentioned in Bölcskei et al. (2017). From this perspective, we see that parameter sharing, mentioned in Section 5.5, is also effective. It can also be said that this paper expands the approximation theory argued in Bölcskei et al. (2017) to estimation theory over sparse parameter spaces.

Thus, the contribution of this paper is summarized as follows:

• •

To deal with sparsity in machine learning, we define function classes $\mathcal{I}_{\Phi}^{0}$ and $\mathcal{K}_{\Psi}^{p}$ with a parameter $p$ controlling the sparsity. We also consider the nonparametric regression problem on these target classes and derive the minimax lower bounds of estimation error.

• •

We consider linear estimators, which are a competitor of deep learning, investigating them by evaluating their estimation error over sparse target classes. We show that linear estimators can only attain suboptimal rates on sparse and non-convex models and even become arbitrarily slow under the same sparsity with other parameters varying. This also gives a unified understanding of existing studies which describe situations where deep learning is superior to linear methods.

• •

To demonstrate the learning ability of the deep ReLU network on sparse spaces, we construct sparse neural networks that nearly attain minimax-optimal rates. It is also shown that parameter sharing in the construction of neural networks is effective on sparse target classes.

We give a brief overview of each section in the following.

In Section 2, we introduce general methods used in statistical learning theory, presenting our own proofs or arguments to the maximum extent possible. Section 2.2 presents an information-theoretic way to obtain a lower bound for the minimax rate. Also, the method for evaluating an estimation error by using an approximation error is given. Evaluations of linear minimax rates are given in Section 3. We prove that linear estimators cannot distinguish between a function class and its convex hull, and as a consequence linear minimax rates can be rather slower than ordinal minimax rates. Section 4 provides the definitions of our own target function classes. The $\ell^{0}$ norm and the $w\ell^{p}$ quasi-norm of coefficients in linear combinations are introduced as indicators of sparsity. The minimax lower bounds for the defined classes are also given (which are revealed to be optimal up to log factors in the section that follows). In Section 5, we show that deep learning attains the nearly minimax rate for defined function classes. In addition, we propose that parameter sharing can be a means of reducing complexities in regularized networks. Finally, Section 6 provides a summary and presents future directions for this work.

1.4 Notation

We use the following notation throughout the paper.

• •

$\|\cdot\|_{\infty}$ and $\|\cdot\|_{0}$ are defined as

[TABLE]

for a vector $v=(v_{1},\ldots,v_{m})^{\top}\in\mathbb{R}^{m}$. They are defined similarly for real matrices.

• •

As a natural extension of $\|\cdot\|_{0}$, $\|a\|_{\ell^{0}}$ denotes the number of nonzero terms in the sequence $a=(a_{i})_{i=1}^{\infty}$.

• •

For $p>0$ and a real sequence $a=(a_{i})_{i=1}^{\infty}$, the $\ell^{p}$ norm of $a$ is defined as

[TABLE]

and $\ell^{p}$ denotes the set of all real sequences with a finite $\ell^{p}$ norm.

2 General theories in statistical estimation

2.1 General settings and notation

Let us consider the following regression model. We observe i.i.d. random variables $(X_{i},Y_{i})$ generated by

[TABLE]

Here, each $\xi_{i}$ is an observation noise independent of other variables. In this paper, we use settings such that each $X_{i}$ is $d$-dimensional and uniformly distributed on $[0,1]^{d}$, each $Y_{i}$ is one-dimensional, and $\xi_{i}$’s are i.i.d. centered Gaussian variables with variance $\sigma^{2}$ ($\sigma>0$). For simplicity, we sometimes use the notation $X^{n}:=(X_{1},\ldots,X_{n})$, $Y^{n}:=(Y_{1},\ldots,Y_{n})$, and $Z^{n}:=(X_{i},Y_{i})_{i=1}^{n}$.

Remark 2.1**.**

In the following, we often write only $\widehat{f}$ to indicate an estimator where we should write $(X_{i},Y_{i})^{n}\mapsto\widehat{f}$ (this mapping is supposed to be measurable). For example, $\inf_{(X_{i},Y_{i})_{i=1}^{n}\mapsto\widehat{f}\in\mathcal{F}}$ is simply denoted by $\inf_{\widehat{f}\in\mathcal{F}}$. In addition, for the case $\mathcal{F}=L^{2}([0,1]^{d})$, we omit $\mathcal{F}$ and simply write $\inf_{\widehat{f}}$.

To evaluate the quality of estimators, we need to adopt some evaluation criteria. For a fixed $f^{\circ}$ and a function $f\in L^{2}([0,1]^{d})$, we have

[TABLE]

This implies that the magnitude of the expected error $\mathrm{E}[(f(X_{i})-Y_{i})^{2}]$ depends only on that of the $L^{2}$ distance $\|f-f^{\circ}\|_{L^{2}}^{2}$. This leads to the following definition for a performance criterion.

Definition 2.2**.**

The $L^{2}$* risk* for an estimator $\widehat{f}$ is defined as

[TABLE]

For a model $\mathcal{F}^{\circ}\subset L^{2}([0,1]^{d})$, the minimax $L^{2}$ risk over $\mathcal{F}^{\circ}$ is defined as

[TABLE]

where $\widehat{f}$ runs over all estimators (measurable functions).

We evaluate the quality of an estimator $\widehat{f}$ by this $L^{2}$ risk and compare it with the minimax-optimal risk.

Remark 2.3**.**

We omit $n$ from the notation because it is treated as a constant when we consider a single regression problem. However, as $n$ goes to $\infty$, the minimax risk converges to [math], and in this paper we are interested in the convergence rate of the minimax risk.

2.2 Relationships between complexity and minimax risk

In this section, we introduce a key procedure to evaluate the minimax risk. To do so, we define complexity measures called $\varepsilon$-entropy ($\varepsilon$ may be taken place by any positive real number), which formally represent complexities of (totally bounded) metric spaces. This kind of complexity of $\mathcal{F}^{\circ}$ profoundly affects the convergence rate of the minimax risk Yang and Barron (1999).

Definition 2.4**.**

van der Vaart and Wellner (1996); Yang and Barron (1999) For a metric space $(S,d)$ and $\varepsilon>0$,

• •

a finite subset $T$ is called $\varepsilon$-packing if $d(x,y)>\varepsilon$ holds for any $x,y\in T$ with $x\neq y$, and the logarithm of the maximum cardinality of an $\varepsilon$-packing subset is called the packing $\varepsilon$-entropy and is denoted by $M_{(S,d)}(\varepsilon)$;

• •

a finite set $U\subset\overline{S}$ is called $\varepsilon$-covering if for any $x\in S$ there exists $y\in U$ such that $d(x,y)\leq\varepsilon$, and the logarithm of the minimum cardinality of an $\varepsilon$-covering set is called the covering $\varepsilon$-entropy and is denoted by $V_{(S,d)}(\varepsilon)$.

Here, $\overline{S}$ is the completion of $S$ with respect to the metric $d$.

The concept of $\varepsilon$-entropy is useful to obtain a lower bound of the minimax risk of some function class $\mathcal{F}^{\circ}$. Let $\mathcal{F}^{\circ}\subset L^{2}([0,1]^{d})$ be the class of true functions, equipped with the $L^{2}$ metric. For simplicity, let $V(\varepsilon)=V_{(\mathcal{F}^{\circ},\|\cdot\|_{L^{2}})}(\varepsilon)$ and $M(\varepsilon)=M_{(\mathcal{F}^{\circ},\|\cdot\|_{L^{2}})}(\varepsilon)$. Then, the following theorem holds.

Theorem 2.5**.**

(Yang and Barron, 1999, Theorem 1)* In the Gaussian regression model, suppose there exist $\delta,\varepsilon>0$ such that*

[TABLE]

Then we have

[TABLE]

where $\mathrm{P}_{f}$ is the probability law with $f^{\circ}=f$, and $\mathrm{E}_{f}$ is the expectation determined by $\mathrm{P}_{f}$.

The above theorem states the relationship between the complexity of the function class and the lower bound of the minimax risk. On the other hand, the following theorem is relates the complexity of the estimator and the upper bound of the generalization error (and the minimax risk, at the same time). The following is also useful for evaluating the convergence rate of the empirical risk minimizer of some explicit model, such as neural networks.

Theorem 2.6**.**

(Schmidt-Hieber, 2017, Lemma 4)* In the Gaussian regression model (1), let $\widehat{f}$ be the empirical risk minimizer, taking values in $\mathcal{F}\subset L^{2}([0,1]^{d})$. Suppose every element $f\in\mathcal{F}$ satisfies $\|f\|_{L^{\infty}}\leq F$ for some fixed $F>0$. Then, for an arbitrary $\delta>0$, if $V_{(\mathcal{F},\|\cdot\|_{L^{\infty}})}(\delta)\geq 1$, then*

[TABLE]

holds, where $C>0$ is an absolute constant.

This sort of evaluation has been obtained earlier in Györfi et al. (2006); Koltchinskii (2006); Giné and Koltchinskii (2006) and the proof for this is essentially the same as earlier ones. For completeness, however, we give a proof for this assertion in the appendix (Section B.1).333We noticed some technical flaws in an earlier version of the proof of Schmidt-Hieber (2017), so we include the proof in the appendix for completeness.

3 Suboptimality of linear estimators

We consider linear estimators as a competitor to deep learning, and in this section, we characterize their suboptimality by the convexity of the target model. Linear estimators, represented by kernel methods, are classically applied to regression problems. Indeed, some linear estimators have minimax optimality over smooth function classes such as Hölder classes and Besov classes with some constraint on their parameters (with fixed design: Donoho and Johnstone, 1998; Tsybakov, 2008). However, as has been pointed out in the literature Korostelev and Tsybakov (1993); Imaizumi and Fukumizu (2019), linear estimators can attain only suboptimal rates with function classes having discontinuity. We here show that the suboptimality of linear estimators arises even with a quite simple target class. Our first contribution is to point out that the concept of the convex hull gives the same explanation to such suboptimality for several target classes, and based on that argument, we then show that linear estimators perform suboptimally even on a quite simple target class.

3.1 Linear estimators and its minimax risk

Definition 3.1**.**

The estimation scheme $(X_{i},Y_{i})_{i=1}^{n}\mapsto\widehat{f}$ is called linear if $\widehat{f}$ has the form

[TABLE]

where we suppose $\mathrm{E}\left[\|\varphi_{i}(\cdot;X^{n})\|_{L^{2}}^{2}\right]<\infty$. Also, we call an estimator $\widehat{f}$ affine if $\widehat{f}$ has the form

[TABLE]

where $\varphi$ has the same condition as $\varphi_{i}$, and $\widehat{f}_{L}$ is a linear estimator.

Remark 3.2**.**

The condition $\mathrm{E}\left[\|\varphi_{i}(\cdot;X^{n})\|_{L^{2}}^{2}\right]<\infty$ may be replaced by a weaker version. This actually assures that

• •

$\varphi_{i}(\cdot;X^{n})\in L^{2}([0,1]^{d})$ holds almost surely;

• •

$\mathrm{E}\left[\varphi_{i}(x;X^{n})^{2}\right]<\infty$ holds almost everywhere.

The latter condition is only needed in the justification of (29).

A linear estimator is of course an affine estimator as well. Linear or affine estimators are classically used often; they include linear (ridge) regression, the Nadaraya–Watson estimator, and kernel ridge regression Tsybakov (2008); Bishop (2006); Friedman et al. (2001). For example, the estimator given by kernel ridge regression can be explicitly written as

[TABLE]

where $\lambda$ is a positive constant, $k:[0,1]^{d}\times[0,1]^{d}\to\mathbb{R}$ is a positive semi-definite kernel, and the matrix $K\in\mathbb{R}^{n\times n}$ is defined as $K:=(k(X_{i},X_{j}))_{i,j}$. We can see that the difference in performance between deep learning and linear estimators becomes large in a non-convex model, which can be explained by the following theorem. (This theorem can also be seen as a generalization of Cai and Low (2004, Theorem 5).)

Let $\operatorname{conv}(\mathcal{F}^{\circ})$ denote the convex hull of $\mathcal{F}^{\circ}$; i.e.,

[TABLE]

Notice that the $\operatorname{conv}(\mathcal{F}^{\circ})$ is larger than the original set $\mathcal{F}^{\circ}$. Let $\overline{\operatorname{conv}}(\mathcal{F}^{\circ})$ be the closure of $\operatorname{conv}(\mathcal{F}^{\circ})$ with respect to the $L^{2}$ metric (caller closed convex hull). The following assertion holds.

Theorem 3.3**.**

For affine methods, the minimax risk over $\mathcal{F}^{\circ}$ coincides with the minimax risk over $\overline{\operatorname{conv}}(\mathcal{F}^{\circ})$; i.e., the following equality holds:

[TABLE]

Proof.

We only prove the assertion

[TABLE]

For the case of $\overline{\operatorname{conv}}(\mathcal{F}^{\circ})$, see the appendix (Section B.2).

Fix an estimator $\widehat{f}$ and let

[TABLE]

For $f^{\circ},g^{\circ}\in\mathcal{F}^{\circ}$ and $t\in(0,1)$, let $h^{\circ}:=tf^{\circ}+(1-t)g^{\circ}$. Then

[TABLE]

holds by Fubini’s theorem (the integrated value is nonnegative). By the convexity of the square, we have

[TABLE]

Here, notice that $\widehat{f}$ is dependent on whether we choose $f^{\circ}$, $g^{\circ}$, or $h^{\circ}$. Therefore, we integrate this inequality to obtain

[TABLE]

This means that $R(\widehat{f},\cdot)$ is a convex functional, and so $\text{LHS}\geq\text{RHS}$ holds in (2). Since it is clear that $\text{LHS}\leq\text{RHS}$, the equality of (2) holds. ∎

Remark 3.4**.**

Indeed, Donoho et al. (1990) and Donoho and Johnstone (1998) pointed out that the convex hull in the above assertion can be replaced by the quadratic hull, which is generally larger than a convex hull in a similar setting (fixed design). However, their propositions require the assumption of fixed design and orthosymmetricity with some wavelet expansion. Hence, we have explicitly noted Theorem 3.3 under milder conditions.

By this theorem, we see that linear estimators hardly achieve the minimax rate in a non-convex model. This also explains the difference between deep learning and linear methods argued in Schmidt-Hieber (2017), Imaizumi and Fukumizu (2019) and Suzuki (2019) in a unified manner. In the following section, we demonstrate a simple example where linear estimator are suboptimal.

3.2 Functions of bounded total variation

Let us consider a specific function class as a simple but instructive example, a class whose convex hull becomes larger in terms of the covering entropy. In addition, the convex hull is dense in $\mathrm{BV}(C)$ (defined below), over which linear estimators can only attain a suboptimal rate.

Definition 3.5**.**

For $k\geq 1$ and $C>0$, define

[TABLE]

as functions from $[0,1]$ to $\mathbb{R}$ with jumps occurring at most $k$ times.

We can also understand $J_{k}(C)$ as the set of piece-wise constant functions.

We next introduce a function class well-known in the field of real analysis, which is indeed related to $J_{k}$’s (Lemma 3.8).

Definition 3.6**.**

For any real numbers $a<b$ and a function $f:[a,b]\to\mathbb{R}$, define the total variation of $f$ on $[a,b]$ as

[TABLE]

Also, for $C>0$, define the set of functions with bounded total variation as

[TABLE]

Remark 3.7**.**

The condition $|f(0)|\leq C$ is needed to bound the size of the set, and it may be replaced by other similar bounding conditions such as $\sup_{t\in[0,1]}|f(t)|\leq C$ or $\int_{0}^{1}|f(t)|\,\mathrm{d}t$ $\leq C$ (e.g., Donoho, 1993). These conditions are equivalent up to constant multiplications of $C$ (i.e., $\mathrm{BV}(C)\subset\mathrm{BV}^{\prime}(\alpha C)\subset\mathrm{BV}(\beta C)$ holds for some $\alpha,\beta>0$, where $\mathrm{BV}^{\prime}$ is a set defined with another constraint). Hence, we adopt $|f(0)|\leq C$ for simplicity of arguments.

Then, we have the following assertion. The proof is given in the appendix (Section B.3)

Lemma 3.8**.**

$\overline{\operatorname{conv}}(J_{k}(C))\supseteq\mathrm{BV}(C)$* holds for each $k\geq 1$ and $C>0$.*

From the above, it follows that linear estimators cannot distinguish $J_{k}(C)$ and $\mathrm{BV}(C)$ in terms of minimax convergence rates (Theorem 3.3).

Since it is known that the unit ball of $B_{1,1}^{1}([0,1])$ is included in $\mathrm{BV}(C)$ for some $C>0$ Peetre (1976), the following theorem can be seen as a special case of Theorem $1$ in Zhang et al. (2002) (see also Table 1).

Theorem 3.9**.**

There exists a constant $c>0$ dependent only on $C$ such that

[TABLE]

holds.

The following corollary is one of the main results in this paper.

Corollary 3.10**.**

For $k=1,2,\ldots$, there exists a constant $c>0$ dependent only on $C$ such that

[TABLE]

holds.

Proof.

The result is clear from Theorem 3.3, Lemma 3.8, and Theorem 3.9. We can of course take the same $c$ as in Theorem 3.9. ∎

Remark 3.11**.**

On the one hand, the minimax-optimal rate of the unit ball of $B^{1}_{1,1}([0,1])$ is $\widetilde{\Theta}(n^{-2/3})$ (Table 1), whereas the counterpart of $J_{k}(C)$ is $\widetilde{\Theta}(n^{-1})$ as is attained by deep learning (proved later; see Corollary 5.8). On the other hand, the fact that the unit ball of $B^{1}_{1,1}([0,1])$ is included in $\mathrm{BV}(C)$ implies that the linear minimax rate of $\mathrm{BV}(C)$ is not faster than that of $B^{1}_{1,1}([0,1])$’s unit ball. Since $\mathrm{BV}(C)$ and $J_{k}(C)$ have the same linear minimax rate, $J_{k}(C)$ is a quite extreme example, even in comparison with $B^{1}_{1,1}([0,1])$.

4 Sparse target function classes

As $J_{k}(C)$’s given in Section 3.2 are too simple, we consider sparse target function classes which are generalizations of $J_{k}(C)$’s and are also related to wavelets. We investigate the performance of deep learning and other methods over these sparse classes. The minimax lower bound for each class is also given by applying the arguments in Section 2.2. Sparsity well characterizes the spaces whose convex hulls are much larger than the original spaces, a property that is essential for the proofs that were given in Section 3.

4.1 The $\ell^{0}$-bounded affine class

The definition of the following class is inspired by the concept of “affine class” treated in Bölcskei et al. (2017).

Definition 4.1**.**

Given a set $\Phi\subset L^{2}([0,1]^{d})$ with $\|\varphi\|_{L^{2}}=1$ for each $\varphi\in\Phi$ along with constants $n_{s}\in\mathbb{Z}_{>0}$ and $C>0$, we define an $\ell^{0}$-bounded affine class $\mathcal{I}_{\Phi}^{0}$ as

[TABLE]

If we adopt a jump-type function $\varphi\in\Phi$ such as $\varphi=1_{[0,1/2)}$, we can see $\mathcal{I}_{\Phi}^{0}(n_{s},C)$ as a generalization of $J_{k}(C)$’s defined in the previous section. The condition for $c_{i}$ is also regarded as $\|c\|_{\ell^{0}}\leq n_{s}$, where the $\ell^{0}$ norm is used as the most extreme measurement of sparsity Raskutti et al. (2011); Wang et al. (2014).

Let us derive a minimax lower bound for this class. Although the proof for this assertion can easily be given by applying the argument appearing in Tsybakov (2008), we provide it in the appendix (Section B.4).

Theorem 4.2**.**

There exists a constant $C_{0}>0$ depending only on $\sigma^{2}$ such that

[TABLE]

holds for each $n\geq 1$.

Remark 4.3**.**

If the set $\Phi$ is simple enough; e.g., a finite set of piece-wise constants or piece-wise polynomials, then this rate can be almost attained by nonlinear estimators. Indeed, in such cases deep learning attains the rate $\mathrm{O}(n^{-1}(\log n)^{3})$ (Theorem 5.6). However, with a single noncontinuous $\varphi\in\Phi$, linear estimators become suboptimal; the linear minimax rate is lower-bounded by $\mathrm{\Omega}(n^{-1/2})$ (Corollary 3.10).

4.2 The $w\ell^{p}$-bounded function classes

The function class we treat in the previous section seems too simple to approximate the real-world data. Therefore, we are going to consider larger function classes with “sparsity”. We introduce concepts for measuring the sparsity of function classes in order to present a simple treatment of several sparse spaces. These concepts were introduced and discussed previously in Donoho (1993), Donoho (1996), and Yang and Barron (1999).

Definition 4.4**.**

For a sequence $a=(a_{i})_{i=1}^{\infty}\in\ell^{2}$, let each $|a|_{(i)}$ denote the $i$-th largest absolute value of terms in $a$. For $0<p<2$, the weak $\ell^{p}$ norm of $a$ is defined as

[TABLE]

Here, notice that $\|\cdot\|_{w\ell^{p}}$ is not a norm, as $(|a|_{(i)})_{i=1}^{\infty}$ is a permutation of $(|a_{i}|)_{i=1}^{\infty}$. However, we call it a “weak $\ell^{p}$ norm” following the notation used in Donoho (1993) and Donoho (1996).

Definition 4.5**.**

Given an orthonormal set $\varphi=(\varphi_{i})_{i=1}^{\infty}\subset L^{2}([0,1]^{d})$ and constants $C_{1},C_{2},\beta>0$ and $0<p<2$, we define a sparse $\ell^{p}$-approximated set $\mathcal{I}_{\varphi}^{p}$ as

[TABLE]

The constraint $\sum_{i=m+1}^{\infty}a_{i}^{2}\leq Cm^{-\beta}$ is called $\beta$-minimally tail compactness of $(a_{i})_{i=1}^{\infty}$, which is required to make the set compact in the $L^{2}$ metric.

Remark 4.6**.**

To represent sparsity, the $\ell^{p}$ norm of coefficients is also used (see, e.g., Raskutti et al., 2011; Wang et al., 2014). Note here that $\|a\|_{\ell^{p}}\leq C$ implies $\|a\|_{w\ell^{p}}\leq C$. Indeed, $\|a\|_{\ell^{p}}\leq C$ means that for each $i$,

[TABLE]

Thus, a weak $\ell^{p}$ ball contains an ordinary $\ell^{p}$ ball. In addition, consider the case in which $d=1$ and $\varphi$ is an orthonormal basis generated by a wavelet in $C^{r}([0,1])$ with $r\in\mathbb{Z}_{>0}$ satisfying $r>\alpha:=1/p-1/2$. Then, the Besov norm $\|\cdot\|_{B^{\alpha}_{p,p}}$ of a function is equivalent to the $\ell^{p}$-norm $\|\cdot\|_{\ell^{p}}$ of wavelet coefficients (Donoho and Johnstone, 1998, Theorem 2). In this case, $\mathcal{I}^{p}_{\varphi}$ may be just a slight expansion of existing space, but our main interest is the case in which $\varphi$ has a discontinuity (e.g., when $\varphi$ is defined by the Haar wavelet), which makes things different. Furthermore, notice that Besov spaces with such parameters are omitted in Table 1 (see the note; the upper bounds are given in Suzuki (2019) for a wider range of parameters, but the range for the given lower bounds for linear estimators is limited).

Hereinafter, we fix $p$, $C_{1}$, $C_{2}$, and $\beta$ and often write $\mathcal{I}^{p}_{\varphi}(C_{1},C_{2},\beta)$ as $\mathcal{I}^{p}_{\varphi}$ if there is no confusion; therefore, constants appearing in the following may depend on these values. In the following arguments, we first derive a minimax lower bound for $\mathcal{I}_{\varphi}^{p}$, and then we introduce a broader function class that is well approximated by neural networks.

To use Theorem 2.5, we exploit the following lemma. As stated in Section 2.2, the covering entropy of the function class is important. The proof of the lemma is given in the appendix (Section B.5).

Lemma 4.7**.**

Let $\alpha:=1/p-1/2$, and suppose $\beta$ satisfies $\beta\leq 2\alpha$. Then there exists a constant $C_{\mathrm{low}},C_{\mathrm{up}}>0$ such that

[TABLE]

holds for each $\varepsilon>0$.

Next, we derive a nearly tight minimax lower bound for $\mathcal{I}_{\varphi}^{p}$. In this case, “nearly” means “up to log factors.”

Theorem 4.8**.**

There exists a constant $C=C(p,C_{1},C_{2})>0$ such that

[TABLE]

holds for each $n\geq 2$.

Proof.

In this proof, we write the $\varepsilon$-entropies of $\mathcal{I}_{\varphi}^{p}$ simply as $V(\varepsilon)$ and $M(\varepsilon)$.

First, let $\displaystyle\varepsilon_{n}:=c\left(\frac{\log n}{n}\right)^{\frac{\alpha}{2\alpha+1}}$ for some constant $c>0$. Then by Lemma 4.7, we have

[TABLE]

where $c_{\mathrm{up}}>0$ is some constant independent of $\varepsilon_{n}$, and we have used $n\geq 2$. Thus we have

[TABLE]

for a sufficiently large $c$.

Second, notice that $M(\varepsilon)\geq V(\varepsilon)$ holds. Indeed, given a maximal $\varepsilon$-packing of $\mathcal{I}_{\varphi}^{p}$, the maximality implies that the set also satisfies the condition for being an $\varepsilon$-covering. Now, let $\delta_{n}:=C^{\prime}n^{-\frac{\alpha}{2\alpha+1}}(\log n)^{-\frac{2\alpha^{2}}{2\alpha+1}}$ for some constant $C^{\prime}>0$. Then we have, by Lemma 4.7,

[TABLE]

for some constant $c_{\mathrm{low}}>0$ independent of $C^{\prime}$, where we have used $n\geq 2$.

By (7), (8), and Theorem 2.5, for a sufficiently small $C^{\prime}$, we have

[TABLE]

and the proof is complete. ∎

Remark 4.9**.**

This minimax lower bound is nearly tight, especially for the wavelet case treated in the following section. Indeed, deep learning (if necessary, with parameter sharing) achieves the rate $\mathrm{O}(n^{\frac{2\alpha}{2\alpha+1}}(\log n)^{3})$, for a broad range of wavelets (Theorem 5.10, 5.14).

4.3 Sparsity conditions for wavelet coefficients

In this subsection, we apply the argument in the previous subsection to orthogonal wavelets. They already have a broad application area from engineering and physics to pure and applied mathematics, especially in signal processing, numerical analysis, and so on Daubechies (1992). Because wavelets are the mathematical model of localized patterns, they are suitable for our motivation in setting function classses (see Section 1.3 and Table 2). For simplicity, we only consider the $1$-dimensional case in this section, and the treatment of multi-dimensional cases is deferred to the appendix (Section A.2).

Definition 4.10**.**

Let $\psi:[0,1]\to\mathbb{R}$ be a function with $\|\psi\|_{L^{2}}=1$. For such a function, we define, for integers $k,\ell$,

[TABLE]

where $\psi$ is treated as [math] outside $[0,1]$. Also, $\psi$ is called an orthogonal wavelet if $\psi$ satisfies

[TABLE]

for all $(k,\ell)\neq(k^{\prime},\ell^{\prime})$.

Definition 4.11**.**

Given a $1$-dimensional orthonormal wavelet $\psi(x)$ and constants $C_{1},C_{2},\beta>0$ and $0<p<2$, define

[TABLE]

For the difinition of multi-dimensional cases, see Definition A.4.

Remark 4.12**.**

If we consider the lexical order on $(k,\ell)$’s, then $\mathcal{J}_{\psi}^{p}(C_{1},C_{2},\beta)$ is revealed to be $\beta$-minimally tail compact (see Definition 4.5). Thus, $\mathcal{I}_{\varphi}^{p}\subset\mathcal{J}_{\psi}^{p}$ holds with some modification of constants and they have the same degree of sparsity.

In the following, we introduce the class $\mathcal{K}^{p}_{\Psi}$ as an expansion of $\mathcal{J}^{p}_{\psi}$. Though the following definition is treating the $d$-dimensional case, considering only the case $d=1$ is sufficient for readers to understand the essential properties. For multi-dimensional cases, see also Section A.2.

Definition 4.13**.**

Let $\Psi\subset L^{2}([0,1]^{d})$ consist of orthonormal wavelets. Then, for an integer $n_{s}>0$ and constants $C_{1},C_{2},C_{3},\beta>0$ and $0<p<2$, define

[TABLE]

Remark 4.14**.**

By Remark 4.12 (and Remark A.5) the bound given in Theorem 4.8 is also the minimax lower bound for $\mathcal{K}_{\Psi}^{p}$. Moreover, $J_{k}(C)$, introduced in Section 3, is included in $K^{p}_{\Psi}$, with $k\leq n_{s}$ and a specific $\Psi$ such as one containing the Haar wavelet. Thus, the bounds given in the previous sections are still applicable to this function class. Indeed, as described in Figure 1, we define $\mathcal{K}^{p}_{\Psi}$ as an expansion of $\mathcal{I}^{0}_{\Psi}$ as well.

4.4 Suboptimlaity of linear estimators on $\mathcal{J}_{\psi}^{p}$ and $\mathcal{K}_{\psi}^{p}$

For a wavelet $\psi$ with compact support, linear estimator shows suboptimality on $\mathcal{J}_{\psi}^{p}$ and $\mathcal{K}_{\psi}^{p}$, as is shown in Theorem 4.15. This can be regarded as a stronger version of Corollary 3.10 and shows the non-efficiency of linear estimators in sparse classes. The proof of this theorem is similar to that of Theorem 1 in Zhang et al. (2002) and is given in the appendix (Section B.6).

Theorem 4.15**.**

Let $d=1$ and $\psi$ be a bounded and compactly supported wavelet. For any constants $C_{1},C_{2},\beta>0$ and $0<p<2$, there exists a constant $C$ dependent only on $C_{1}$, $C_{2}$, and $\beta$ such that

[TABLE]

holds for each $n\geq 1$. This bound also holds on $\mathcal{K}_{\Psi}^{p}$.

Remark 4.16**.**

From this result, we see that the minimax-optimal rate for linear estimators can be arbitrarily slow even with the same sparsity $p$, i.e., with a bounded covering entropy (by Lemma 4.7). The nearly optimal rates attained by deep learning given in Section 5 are unfortunately limited to the case $\beta>1$ (because of the assumption of boundedness), but this still serves as evidence for the non-effectiveness of linear methods in estimating sparse classes.

5 Learning ability of deep ReLU neural networks

5.1 Mathematical formulation of deep ReLU neural networks

For mathematical treatments of neural networks, we have referenced some recent papers on approximation theory and estimation theory Suzuki (2019); Schmidt-Hieber (2017); Yarotsky (2017); Bölcskei et al. (2017); Keiper et al. (2017). In the following, we define neural networks mathematically and evaluate their covering entropies.

Definition 5.1**.**

Let $\rho:\mathbb{R}\to\mathbb{R}$. For $L,S,D\in\mathbb{Z}_{>0}$ and $B\geq 1$ (with $D\geq d$), define $\mathcal{N}(L,S,D,B)$ as the set of all functions $f:\mathbb{R}^{d}\to\mathbb{R}$ of the form

[TABLE]

satisfying

[TABLE]

and

[TABLE]

where $\rho$ is operated elementwise, and $L$, $S$, and $D$ denote the number of hidden layers, the sparsity, and the dimensionality of the layers, respectively. Also, for $F>0$, we consider a function class

[TABLE]

5.2 Complexity of neural network models

Hereinafter, we use the ReLU activation function $\rho(x)=\max\{x,0\}$. Notice that $\mathcal{N}_{F}$ can be realized easily using ReLU activation after an element of $\mathcal{N}$ is computed. To evaluate the generalization error of deep learning, we need bounds for the complexity of neural network models. The following lemma gives the required bound as a function of parameters $L,S,D,B$.

Lemma 5.2**.**

Schmidt-Hieber (2017); Suzuki (2019)* For any $0<\delta<1$, the $\delta$-covering entropy with respect to $\|\cdot\|_{L^{\infty}}$ of $\mathcal{N}(L,S,D,B)$ (limiting the domain to $[0,1]^{d}$) can be bounded as*

[TABLE]

The proof for this lemma is given in the appendix (Section B.7). We next introduce a lemma which evaluates the approximation ability of a neural network given the approximation ability of subnetworks. This lemma is also stated in Bölcskei et al. (2017) in another form. Thanks to this lemma, we can only consider the approximation of the wavelet basis, where we can exploit many existing studies.

Lemma 5.3**.**

Let $\Phi\subset L^{2}([0,1]^{d})$ satisfy $\|\varphi\|_{L^{2}}=1$ for each $\varphi\in\Phi$. Suppose for any $\varphi\in\Phi$ there exists a function $g\in\mathcal{N}(L,S,D,B)$ such that $\|g-\varphi\|_{L^{2}}\leq\varepsilon$. Then for any $f^{\circ}\in\mathcal{I}^{0}_{\Phi}(n_{s},C)$, there exists a function

[TABLE]

such that $\|f-f^{\circ}\|_{L^{2}}\leq C^{3/2}n_{s}\varepsilon$ holds.

Proof.

The approximation of $f^{\circ}(x)=\sum_{i=1}^{n_{s}}c_{i}\varphi_{i}(A_{i}x-b_{i})\in\mathcal{I}_{\Phi}^{0}(n_{s},C)$ can be constructed as shown in Fig. 4 (we use the ReLU activation function, and so we compute $\max\{\widetilde{\varphi}_{i},0\}$ and $\max\{-\widetilde{\varphi}_{i},0\}$ and combine them afterward), where each $\widetilde{\varphi}_{i}$ approximates $\varphi_{i}$ with an $L^{2}$-error of at least $\varepsilon$. In this construction,

[TABLE]

holds. ∎

Remark 5.4**.**

These lemmas evaluate the covering entropy and approximation ability of neural networks. Given these two informations, we can exploit Theorem 2.6 to estimate the generalization error of deep learning by a fixed size of neural network. However, we should know how large the parameters $L,S,D,B$ get as $\varepsilon$ goes to zero (or the sample size $n$ goes to infinity). As one can easily see, a shallow ReLU network can approximate piece-wise constant functions easily (more precisely, see e.g., Fig. 4 and the proof of Corollary 5.8). Moreover, deep ReLU networks can approximate polynomials very efficiently Yarotsky (2017). Combining these, for example, we can state that

there exists an architecture $\mathcal{N}(L,S,D,B)$ with $L,S,D=\mathrm{O}(\log(1/\varepsilon))$ and $B=\mathrm{O}(1/\varepsilon)$ that can approximate piece-wise polynomials with $\mathrm{O}(\varepsilon)$ $L^{2}$-error.

Rigorously speaking, we of course have to restrict the function class (the number of non-smooth points, the degree of polynomials, the magnitude of coefficients, etc). However, as the main concern of this paper is statistical viewpoints, we do not get deeper. The important thing is that the above size of network architecture is reasonable to approximate “cheap” functions.

Motivated by the above remark, we define the function class $\mathrm{AP}$, which can be approximated by “light” networks, as follows.

Definition 5.5**.**

For $C_{1},C_{2}>0$, define $\mathrm{AP}(C_{1},C_{2})$ as the set of all functions $\varphi\in L^{2}([0,1]^{d})$ satisfying that, for each $0<\varepsilon<1/2$, there exist parameters $L_{\varepsilon},S_{\varepsilon},D_{\varepsilon},B_{\varepsilon}>0$ such that

• •

$L_{\varepsilon},S_{\varepsilon},D_{\varepsilon}\leq C_{1}\log(1/\varepsilon)$ and $B_{\varepsilon}\leq C_{2}/\varepsilon$ hold;

• •

there exists a $\widetilde{\varphi}\in\mathcal{N}(L_{\varepsilon},S_{\varepsilon},D_{\varepsilon},B_{\varepsilon})$ such that $\|\widetilde{\varphi}-\varphi\|_{L^{2}}\leq\varepsilon$.

5.3 Generalization ability for an extreme case ($\mathcal{I}_{\Phi}^{0}$)

The learning ability of neural networks over $\mathcal{I}_{\Phi}^{0}$ is shown in the following. This is the most extreme case in terms of the difference between the performance of deep learning and linear methods.

Theorem 5.6**.**

Let $\Phi\subset L^{2}([0,1]^{d})$ satisfy $\|\varphi\|_{L^{2}}=1$ for each $\varphi\in L^{2}([0,1]^{d})$ and $\sup_{\varphi\in\Phi}\|\varphi\|_{L^{\infty}}<\infty$. Suppose also $\Phi\subset\mathrm{AP}(C_{1},C_{2})$ holds for some constants $C_{1},C_{2}>0$. Then, for each $n_{s},C>0$, there exist constants $F,C_{3}>0$ dependent only on $n_{s},C$ (independent of $n$) such that the empirical risk minimizer $\widehat{f}$ over $\mathcal{N}_{F}^{(n)}$ satisfies

[TABLE]

for $n\geq 2$, where $\mathcal{N}_{F}^{(n)}$ denotes

[TABLE]

Remark 5.7**.**

This theorem implies that the empirical risk minimizer over $\mathcal{N}(L,S,D,B)$ with size

[TABLE]

almost attains the minimax-optimal rate. Therefore, from the viewpoint of real-world application, a reasonable size of network can behave optimally.

Proof of Theorem 5.6.

Let $M:=\sup_{\varphi\in\Phi}\|\varphi\|_{L^{\infty}}$. If we define $F:=n_{s}CM$, each $f^{\circ}\in\mathcal{I}_{\Phi}^{0}(n_{s},C)$ satisfies $\|f\|_{L^{\infty}}\leq F$. Indeed, $f^{\circ}$ has some expression $f^{\circ}=\sum_{i=1}^{n_{s}}c_{i}\varphi_{i}(A_{i}\cdot-b_{i})$, and so we have

[TABLE]

Hence, for $f\in\mathcal{N}(L,S,D,B)$ and $f^{\circ}\in\mathcal{I}_{\Phi}^{0}(n_{s},C)$, $\widetilde{f}:=\operatorname{sgn}(f)\min\{|f|,F\}$ satisfies

[TABLE]

Also, notice that $\mathcal{N}_{F}$’s covering entropy is not greater than that of $\mathcal{N}$, and so we have, by Lemma 5.2 and the assumption of the assertion,

[TABLE]

for some constant $C_{0}>0$. Then, by Theorem 2.6 and Lemma 5.3,

[TABLE]

holds for some $C_{3}>0$. ∎

Corollary 5.8**.**

Let $d=1$. For $J_{k}(C)$ in Definition 3.5, there exist a constant $F>0$ and a sequence of neural networks $(\mathcal{N}^{(n)})_{n=2}^{\infty}$ such that the empirical risk minimizer $\widehat{f}$ satisfies

[TABLE]

for some constant $C_{3}>0$ independent of $n$ and each $n\geq 2$.

Proof.

By Theorem 5.6, it suffices to show that $\varphi=\sqrt{2}\cdot 1_{[1/2,1]}$ can be approximated within $\varepsilon$-error in $L^{2}$ by a neural network satisfying the condition of Theorem 5.6. This can be actually realized by a shallow network, as

[TABLE]

(Fig. 4) satisfies the desired condition. ∎

Remark 5.9**.**

By Corollary 3.10, Theorem 4.2, and Corollary 5.8, $J_{k}(C)$ demonstrates an extreme situation, wherein neural network learning attains the optimal rate up to $\log$ factors whereas linear methods are suboptimal.

This result can easily be expanded to the case of $d=2$ (if we properly define $J_{k}$ for higher dimensions). In addition, we can treat a set broader than $J_{k}(C)$ as $\mathcal{I}_{\Phi}^{0}$ because smooth functions such as polynomials can be well approximated by $\mathrm{O}(\log(1/\varepsilon))$ weights as has been mentioned in Remark 5.4.

5.4 Generalization ability for the wavelet case ($\mathcal{J}_{\psi}^{p}$, $\mathcal{K}_{\Psi}^{p}$)

Let us consider the case in which the target function class is $\mathcal{K}_{\Psi}^{p}$ in Definition 4.13. Note that, by Lemma 5.3, we only have to consider approximating functions in $\mathcal{J}_{\psi}^{p}$ in Definition 4.11 for fixed $n_{s}$. We defer the proof of the following main result to the appendix (Section B.8).

Theorem 5.10**.**

Let $\psi\in L^{2}([0,1]^{d})$ be an orthonormal wavelet. Suppose there exist constants $C_{1}^{\prime},C_{2}^{\prime}>0$ such that $\psi\in\mathrm{AP}(C_{1}^{\prime},C_{2}^{\prime})$ holds. Then, for each $C_{1},C_{2},\beta>0$ and $0<p<2$, there exists a constant $C>0$ dependent only on constants $C_{1}^{\prime},C_{2}^{\prime},p,C_{1},C_{2},$ $\beta$ (independent of $n$) such that the empirical risk minimizer $\widehat{f}$ over $\mathcal{N}_{F}^{(n)}$ (with some network architecture) satisfies

[TABLE]

for each $F\geq\max\{1,\sigma\}$ and $n\geq 2$, where $\alpha:=1/p-1/2$. Moreover, for a set of wavelets $\Psi\subset\mathrm{AP}(C_{1}^{\prime},C_{2}^{\prime})$, the same evaluation is valid for $\mathcal{K}^{p}_{\Psi}$;

[TABLE]

holds for some $C^{\prime}>0$, where $\widehat{f}$ is the empirical risk minimizer of some larger network.

Remark 5.11**.**

The actual network size of $\mathcal{N}(L,S,D,B)$ here is

[TABLE]

(see (37) in the appendix). Though this network is larger than that of Remark 5.7, this is still acceptable size, especially with large $\alpha$; i.e., in the case the parameter space is very sparse.

Concretely, $\psi$ constructed by using the Haar wavelet satisfies the desired condition. Also, in the case of $d=1$ and $\beta>1$, we can remove the restriction by the constant $F$, because $\sup\{\|f\|_{\infty}\mid f\in J^{p}_{\psi}\}<\infty$ holds. Note also that this result is nearly minimax optimal; i.e., $\widehat{f}$ attains the minimax lower bound derived in Theorem 4.8 up to $\log$ factors and the constraint of boundedness. As $p$ gets smaller, $\mathcal{K}_{\Psi}^{p}$ gets sparser and more reasonable size of neural network can achieve the minimax optimality. Moreover, as is stated in the next remark, linear estimators cannot exploit the sparsity of $\mathcal{K}_{\Psi}^{p}$ in typical settings (e.g., when $\Psi$ contains non-continuous functions). From the viewpoint of approximation ability, this situation can be interpreted as follows; neural networks are allowed to approximate just the sparse set $\mathcal{K}_{\Psi}^{p}$, whereas linear estimators must be able to approximate all the functions contained in $\operatorname{conv}(\mathcal{K}_{\Psi}^{p})$. This difference yields the difference of complexity; linear estimators require far more parameters than neural networks. Finally, this situation results in the great difference of generalization ability between deep and linear.

Remark 5.12**.**

If $\Psi$ contains the Haar wavelet and $C_{3}$ is sufficiently large, $\mathcal{K}_{\Psi}^{p}$ includes $J_{k}$ in Definition 3.5 (also, notice that $J_{k}$ is bounded in the $L^{\infty}$-norm sense). To be more precise, $\{f\in[0,1]^{d}\to\mathbb{R}\mid f(x_{1},\ldots,x_{d})=g(x_{1}),\ g\in J_{k}(C)\}$ is included in $K_{\Psi}^{p}$ for some $C>0$. The proof of Theorem 3.9 can easily be modified for this case, and we have

[TABLE]

for some $c>0$. If $\alpha>1/2$ (equivalent to $p<1$) holds, then the neural network learning is superior to linear methods.

5.5 Parameter sharing technique to restrict the covering entropy

The assumption of the ability for $\varphi$ to be approximated by $\widetilde{\varphi}$ imposed in Theorems 5.6 and 5.10 is quite strong, and thus we cannot treat a broad range of wavelets. In the proof of Theorem 5.10, however, we do not exploit the full degree of freedom depicted in Fig. 4 because subnetworks share the same approximator $\widetilde{\psi}$. In this subsection, we consider neural networks with parameter sharing.

Definition 5.13**.**

Let $N$ be a positive integer. For a given neural network architecture $\mathcal{N}(L,S,D,B)$, denote the $N$-sharing of $\mathcal{N}(L,S,D,B)$ by $\mathcal{N}^{N}(L,S,D,B)$, defined as

[TABLE]

Theorem 5.14**.**

Given a positive integer $N$ and $\mathcal{N}(L,S,D,B)$ with $L\geq 2$, the $\delta$-covering entropy with respect to $\|\cdot\|_{L^{\infty}}$ of $\mathcal{N}^{N}(L,S,D,B)$ (limiting the domain to $[0,1]^{d}$) can be bounded as

[TABLE]

for any $0<\delta<1$.

The proof is straightforward but a bit technical; thus, we defer it to the appendix (Section B.9).

Remark 5.15**.**

If we use neural networks with parameter sharing, we can use non-trivial wavelets with some smoothness; i.e., in Theorem 5.10, the assumption for $\psi$ can be weakened to the following:

For each $0<\varepsilon<1/2$, there exist $L_{\varepsilon},S_{\varepsilon},D_{\varepsilon},B_{\varepsilon}>0$ satisfying

• •

$L_{\varepsilon}\leq C_{1}^{\prime}\log(1/\varepsilon)$, $D_{\varepsilon},B_{\varepsilon}\leq C_{2}^{\prime}\varepsilon^{-\gamma}$ and $S_{\varepsilon}\leq C_{3}^{\prime}\varepsilon^{-\frac{1}{\alpha+1}}$ hold for some constants $C_{1}^{\prime},C_{2}^{\prime},C_{3}^{\prime},\gamma>0$;

• •

there exists $\widetilde{\psi}\in\mathcal{N}(L_{\varepsilon},S_{\varepsilon},D_{\varepsilon},B_{\varepsilon})$ such that $\|\widetilde{\psi}-\psi\|_{L^{2}}\leq\varepsilon$.

This class of $\psi$ is actually broadened as there exist compactly supported wavelets with high regularity (a large Hölder exponent) Daubechies (1992), and such functions can be approximated well by networks with a small number of parameters. Indeed, Yarotsky (2017) proved that

A unit ball of Sobolev space $W^{n,\infty}([0,1]^{d})$ can be approximated, with $L^{\infty}$-error at most $\varepsilon$, by a neural network with $\mathrm{O}(\log(1/\varepsilon))$ depth, and $\mathrm{O}(\varepsilon^{-d/n})$ complexity (corresponds to $S,D$).

Though we cannot use this in the original form because $L^{\infty}$ should be replaced by $L^{2}$ and the upper bound of $B$ should be given, the refinement is not so difficult (e.g., Schmidt-Hieber, 2017).

6 Summary and discussion

6.1 Summary

In this paper, we have shown that deep learning outperforms other commonly used methods such as linear estimators even in a simple case. To evaluate the learning ability of estimators, we employed a Gaussian regression problem with a sparse target function space. In such a problem setting, neural network learning attains nearly the minimax-optimal rate of convergence with respect to the sample size, whereas a linear estimator can only achieve a suboptimal rate. The main novelty is that the target function spaces were selected to have natural sparsity, instead of following the well-known settings developed by the existing mathematical analyses. We have also shown that parameter sharing is quite effective for widening function classes where (near) minimax optimality holds.

6.2 Discussion and future work

There are two main limitations in this work that remain to be addressed in future investigations.

First, $\mathcal{I}^{0}_{\Phi}(n_{s},C)$ is the most extreme case in the sense that deep learning outperforms linear estimators. This class is very simple, and we have additionally defined $w\ell^{p}$-bounded classes $\mathcal{K}^{p}_{\Psi}$ for $0<p<2$, with the assumption of orthonormal wavelets. However, we should remove the orthogonality if we follow the philosophy of defining $\mathcal{I}^{0}_{\Phi}(n_{s},C)$. For example, using the definition

[TABLE]

would be one possible way. Of course, for some range of $(p,\beta)$, we can show that deep learning attains a rate faster than do linear estimators. However, we could not have shown that the convergence rate satisfies minimax optimality, even up to log factors. This difficulty arises from the fact that we have fully exploited the orthogonality in the proof of deep learning’s minimax optimality over $\mathcal{K}^{p}_{\Psi}$. It is possible that we can find both a better minimax lower bound for $\mathcal{L}_{\Phi}^{p}$ and a better approximation bound by neural networks.

Second, parameter sharing, mentioned in Subsection 5.5, is used mainly in the context of convolutional neural networks (CNNs), and this implies the superiority of CNNs in solving a regression problem. However, some of the arguments in this paper are not directly applicable to the analysis of CNNs. Although CNNs have achieved notable success in pattern recognition, theories of CNNs with respect to regression problems have not yet been well argued in the literature.

In addition to these issues, a theoretical analysis of stochastic optimization as used in deep learning is needed, which is not treated in this paper.

Acknowledgments

The authors would like to thank Atsushi Nitanda for his constructive feedback. TS was partially supported by MEXT Kakenhi (15H05707, 18K19793 and 18H03201), Japan Digital Design, and JST-CREST.

Appendix A Mathematical supplements

A.1 Generalities related to the minimax risk

In the Gaussian regression model, for two true functions $f,g\in\mathcal{F}^{\circ}$, it is well known that the Kullback–Leibler (KL) divergence between two distributions of $(X,Y)$ generated by $f$ and $g$ is easy to calculate. Let $d_{\mathrm{KL}}(f,g)$ be the square root of the KL divergence. The following lemma is essential when one treats a regression problem as a parameter estimation problem Yang and Barron (1999); Schmidt-Hieber (2017); Suzuki (2019); its proof is given in Section B.10.

Lemma A.1**.**

It holds that $\displaystyle d_{\mathrm{KL}}(f,g)^{2}=\frac{1}{2\sigma^{2}}\|f-g\|_{L^{2}}^{2}$.

Therefore, the square root of the KL divergence is a metric equivalent to the $L^{2}$ metric in regression problems with a Gaussian noise.

We also introduce a lemma that is useful for deriving a lower bound of the metric entropy.

Lemma A.2**.**

(Donoho, 1993, Lemma 4)* Let $\mathcal{C}_{k}\subset\ell^{2}$ be a $k$-dimensional hypercube of side $2\delta>0$ defined as*

[TABLE]

Then there exists a constant $A>0$ such that

[TABLE]

A.2 Treatment of multi-dimensional wavelets

In this section, we give a multi-dimensional extension of arguments in Section 4.3. Here, we adopt the multiplication of elementwise wavelets as a multi-dimensional wavelet.

Lemma A.3**.**

For orthogonal wavelets $\psi^{(1)},\ldots,\psi^{(d)}$,

[TABLE]

is a $d$-dimensional orthogonal wavelet; i.e.,

[TABLE]

is an orthonormal subset of $L^{2}([0,1]^{d})$.

Proof.

The normality is clear by Fubini’s theorem. Also, for distinct wavelets $\psi,\psi^{\prime}$ in the set, there exists $i$ such that $(k_{i},\ell_{i})\neq(k_{i}^{\prime},\ell_{i}^{\prime})$. Since we have $\psi\psi^{\prime}\in L^{1}([0,1]^{d})$ by the AM-GM inequality, Fubini’s theorem leads to the conclusion. ∎

Definition A.4**.**

Given an orthonormal wavelet

[TABLE]

and constants $C_{1},C_{2},\beta>0$ and $0<p<2$, define

[TABLE]

where the sets $T_{m}$ ($m=0,1,\ldots$) are defined as

[TABLE]

and $\psi_{k,\ell}$ denotes $\displaystyle\prod_{i=1}^{d}\psi_{k_{i},\ell_{i}}^{(i)}$ for each

[TABLE]

Remark A.5**.**

If we define a partial order on $S_{0}$ by $(k,\ell)\preceq(k^{\prime},\ell^{\prime})\Leftrightarrow\max_{i}k_{i}\leq\max_{i}k_{i}^{\prime}$ and then sort it, $\mathcal{J}_{\psi}^{p}(C_{1},C_{2},\beta)$ is revealed to be $\beta/d$-minimally tail compact. Thus, $\mathcal{I}_{\varphi}^{p}\subset\mathcal{J}_{\psi}^{p}$ holds with some modification of constants.

Appendix B Proofs

B.1 Proof of Theorem 2.6

(mainly following the original proof) First, we evaluate the value of

[TABLE]

Let $X_{1}^{\prime},\ldots,X_{n}^{\prime}$ be i.i.d. random variables generated to be independent of $(X_{i},Y_{i})_{i=1}^{n}$. Then we have

[TABLE]

and so we obtain

[TABLE]

Here, let $G_{\delta}=\{f_{1},\ldots,f_{N}\}$ be a $\delta$-covering of $\mathcal{F}$ with the minimum cardinality in the $L^{\infty}$ metric. Notice that $\log N\geq 1$. If we define $g_{j}(x,x^{\prime}):=(f_{j}(x)-f^{\circ}(x))^{2}-(f_{j}(x^{\prime})-f^{\circ}(x^{\prime}))^{2}$ and a random variable $J$ taking values in $\{1,\ldots,N\}$ such that $\|\widehat{f}-f_{J}\|_{L^{\infty}}\leq\delta$, we have

[TABLE]

In the above evaluation, we have used the inequality

[TABLE]

Define constants $r_{j}:=\max\{A,\|f_{j}-f^{\circ}\|_{L^{2}}\}$ ($j=1,\ldots,N$) and a random variable

[TABLE]

where $A>0$ is a deterministic quantity fixed afterward. Then, because of (9), we have

[TABLE]

by the Cauchy–Schwarz inequality and the AM-GM inequality. Here, by the definition of $J$, $\mathrm{E}[r_{J}^{2}]$ can be evaluated as follows:

[TABLE]

Because of the independence of the defined random variables,

[TABLE]

holds, where we have used the fact that each $g_{j}(X_{i},X_{i}^{\prime})$ is centered. Then, using Bernstein’s inequality, we have, in terms of $r:=\min_{1\leq j\leq N}r_{j}$,

[TABLE]

Let us evaluate $\mathrm{E}[T^{2}]$. For arbitrary $t_{0}>0$, it holds that

[TABLE]

We compute the values of these two integrals in terms of $t_{0}$:

[TABLE]

Now we determine $A=\sqrt{t_{0}}/6n$. Since we have $r\geq A=\sqrt{t_{0}}/6n$,

[TABLE]

holds. Letting $t_{0}=8F^{2}n\log N$, the above evaluation can be rewritten as

[TABLE]

Finally, we combine (10), (11), (12), and $\displaystyle A^{2}=\frac{2F^{2}\log N}{9n}$ to obtain

[TABLE]

where we have used the fact that $\log N\geq 1$. Thus, we obtain the evaluation

[TABLE]

Next, we evaluate the quantity

[TABLE]

Since $\widehat{f}$ is an empirical risk minimizer, for arbitrary $f\in\mathcal{F}$,

[TABLE]

holds. As $Y_{i}=f^{\circ}(X_{i})+\xi_{i}$, we have

[TABLE]

Here we have used the fact that

[TABLE]

holds because of the independence between $\xi_{i}$ and $X_{i}$ and the fact that both $\xi_{i}$ and $f(X_{i})$ have a finite $L^{1}$ norm. Thus we have

[TABLE]

Let us evaluate the second term on the right-hand side.

[TABLE]

Here, the first term is upper-bounded by applying the Cauchy–Schwarz inequality:

[TABLE]

Let $\varepsilon_{j}$ $(j=1,\ldots,N)$ be random variables defined as

[TABLE]

where $\varepsilon_{j}:=0$ if the denominator equals [math]. Notice that each $\varepsilon_{j}$ follows a centered Gaussian distribution with variance $\sigma^{2}$ (conditional on $X_{1},\ldots,X_{n}$). Now we have, using the Cauchy–Schwarz inequality and the AM-GM inequality,

[TABLE]

By a similar argument as given in the proof of Lafferty et al. (2008, Theorem 7.47), for any $0<t<1/2\sigma^{2}$,

[TABLE]

holds. Therefore we have, by determining $t=1/4\sigma^{2}$,

[TABLE]

Now we combine (15)–(19) to obtain

[TABLE]

and so

[TABLE]

holds.

Finally, since $f$ is an arbitrary element of $\mathcal{F}$, we combine (13), (14), and (20) to have

[TABLE]

and this leads to the conclusion.

B.2 Proof of Theorem 3.3

We first prove the following lemma.

Lemma B.1**.**

For any affine estimator $\widehat{f}$ and a sequence $f^{\circ}_{1},f^{\circ}_{2},\ldots\in L^{2}([0,1]^{d})$ convergent to $f_{\infty}^{\circ}\in L^{2}([0,1]^{d})$ almost everywhere,

[TABLE]

holds.

Proof.

The conclusion follows immediately from Fatou’s lemma and Eqs. (4) and (5). ∎

Remark B.2**.**

In the proof of Lemma B.1, we have not used the linearity of $\widehat{f}$. Indeed, if $\widehat{f}(X)-f^{\circ}_{m}(X)$ is convergent to $\widehat{f}(X)-f^{\circ}_{\infty}(X)$ with probability $1$, then we have the same conclusion (where $X$ is a uniformly distributed random variable independent of other observed random variables). Hence, Lemma B.1 is applicable to a broader class of estimators, such as estimators continuous with respect to observed data in some metric.

The assertion of Theorem 3.3 is now clear from Eq. (2), Lemma B.1, and the fact that a sequence convergent to a function in $L^{2}$ has a subsequence that is convergent to the same function almost everywhere.

B.3 Proof of Lemma 3.8

The following lemma is a well-known property of functions of bounded total variation.

Lemma B.3**.**

Stein and Shakarchi (2005)* For each function $f:[0,1]\to\mathbb{R}$ with $TV^{f}([0,1])<\infty$, there exist increasing $f^{+},f^{-}:[0,1]\to\mathbb{R}$ such that*

[TABLE]

holds.

By using this lemma, Lemma 3.8 can be proven as follows.

Since $J_{1}(C)\subset J_{k}(C)$ holds for each $k$, it suffices to show the assertion for $k=1$. By the definition of convex hull, we have

[TABLE]

It is obvious that $J_{k}(C)\subset BV(C)$ for each $k$. Thus, we have only to show that for each $f\in BV(C)$ and $\varepsilon>0$, there exist some $k\geq 1$ and $f_{k}\in J_{k}(C)$ such that $\|f_{k}-f\|_{L^{2}}\leq\varepsilon$ holds.

Let $f\in BV(C)$, and take $f^{+}$ and $f^{-}$ satisfying the condition of Lemma B.3. Then $f$ can be written as $f=a_{0}+f^{+}-f^{-}$, where $a_{0}:=f(0)$ is a constant. Let $g:[0,1]\to\mathbb{R}$ be an increasing function satisfying $g(0)=0$, and define

[TABLE]

Then we have $g_{k}\in J_{k}(g(1))$ and $g_{k}(t)=g(i/k)$ for $t\in[i/k,(i+1)/k)$, and so

[TABLE]

holds because $g$ is increasing. Take $f^{+}_{k},f^{-}_{k}$ similarly, and $f_{k}:=a_{0}+f^{+}_{k}-f^{-}_{k}$ satisfies

[TABLE]

and the proof is complete. We can of course take $f_{k}$ directly only from $f$, but we have chosen an easier argument.

B.4 Proof of Theorem 4.2

Since a smaller set makes the minimax risk smaller, it suffices to consider

[TABLE]

For simplicity, assume $C\geq 1$ (otherwise, take the functions and constants appearing below to be smaller at that rate). Define for each $n=1,2,\ldots$

[TABLE]

Then, by applying the argument appearing in Yang and Barron (1999), we have

[TABLE]

where $\widetilde{f}$ is the closer of the two ($f^{+}_{n},f^{-}_{n}$) to $\widehat{f}$. Following the argument in Tsybakov (2008, Proposition 2.1), we have for any $t>0$

[TABLE]

Here, $Z^{n}$ denotes the i.i.d. sequence $(X_{i},Y_{i})_{i=1}^{n}$, and $\,\mathrm{d}\mathrm{P}_{f^{+}_{n}}/\,\mathrm{d}\mathrm{P}_{f^{-}_{n}}$ represents the Radon–Nikodym derivative. Then we have, by (21) and (22),

[TABLE]

When $f^{\circ}=f^{-}_{n}$ holds, the Radon–Nikodym derivative appearing in (23) can be explicitly written as

[TABLE]

Let us consider the right-hand side of (24). First, $\sum_{i=1}^{n}\xi_{i}\varphi_{1}(X_{i})$ is a sum of independent symmetric random variables, and so the sum itself is also symmetric ($X$ is symmetric if $-X$ has the same distribution as $X$), and thus we have $\mathrm{P}\left(\sum_{i=1}^{n}\xi_{i}\varphi_{1}(X_{i})\geq 0\right)\geq 1/2$. Second, for any $s>0$, by Markov’s inequality,

[TABLE]

holds. We determine $s=2/\sigma^{2}$ to obtain the evaluation

[TABLE]

Finally, we have

[TABLE]

By (23)–(25) and letting $t=e^{-2/\sigma^{2}}$, we have

[TABLE]

and so we finally obtain the evaluation

[TABLE]

and this is the desired result.

B.5 Proof of Lemma 4.7

First, we prove the lower-bound part, partially using the proofs in Donoho (1996). When we consider the covering entropy, we have only to consider the coefficients, as $\varphi=(\varphi_{i})_{i=1}^{\infty}$ is an orthonormal set. Thus, define $\mathcal{A}\subset\ell^{2}$ as

[TABLE]

Then it suffices to evaluate $V(\varepsilon):=V_{(\mathcal{A},\|\cdot\|_{\ell^{2}})}(\varepsilon)$.

For each $k=1,2,\ldots$, let $a^{(k)}\in\ell^{2}$ be defined as

[TABLE]

Then $\|a^{(k)}\|_{w\ell^{p}}=C_{1}$ holds. Let us consider the second condition. For $1\leq m\leq k$,

[TABLE]

holds. Since $x^{\beta}(k-x)$ is maximized over $x\in[0,k]$ at $x=\frac{\beta}{1+\beta}k$, the left-hand side of (27) is bounded independent of $m$ as

[TABLE]

where we have used the assumption $\beta\leq 2\alpha$ for the latter inequality. If we define a constant

[TABLE]

then each $Ca^{(k)}$ is an element of $\mathcal{A}$. For each $k$, let us consider a hyperrectangle defined as

[TABLE]

Obviously, each $\mathcal{A}_{k}$ is a subset of $\mathcal{A}$ (this actually is based on the fact that $\varphi$ is an unconditional basis of $\mathcal{I}_{\varphi}^{p}$), and so we have $V(\varepsilon)\geq V_{(\mathcal{A}_{k},\|\cdot\|_{\ell^{2}})}(\varepsilon)$. For each pair of distinct vertices of $\mathcal{A}_{k}$ (which has $2^{k}$ vertices), the $\ell^{2}$ distance between the two is at least $CC_{1}k^{-1/p}$, and so, by setting $\delta=k^{-1/p}$ in Lemma A.2, we have, for $A$ appearing in the lemma,

[TABLE]

for each $k$. If we write $C^{\prime}=CC_{1}/2$, then we have $V(C^{\prime}k^{-\alpha})\geq Ak$. Therefore, for $\varepsilon\in[C^{\prime}2^{-(j+1)\alpha},C^{\prime}2^{-j\alpha}]$,

[TABLE]

holds, where $c:=AC^{\prime 1/\alpha}/2$. This evaluation holds only for $0<\varepsilon\leq C^{\prime}$; however, we have $V(\varepsilon)\geq 1$ for $\varepsilon>C^{\prime}$, and hence $V(\varepsilon)\geq C^{\prime 1/\alpha}\varepsilon^{-1/\alpha}$ holds. Thus, we have reached the desired result.

Then, we deal with the upper-bound part. By the same logic as in the proof of lower-bound, it suffices to evaluate the metric entropy of $\mathcal{A}$ in (26). Let $V(\varepsilon):=V_{(\mathcal{A},\|\cdot\|_{\ell^{2}})}(\varepsilon)$ similarly. For an arbitrary element $a\in\mathcal{A}$, let $i_{j}$ be the index of the term of $a$ having the $j$-th largest absolute value; i.e.,

[TABLE]

which is a permutation of $(|a_{i}|)_{i=1}^{\infty}$. By (6) and the definition of $\mathcal{A}$,

[TABLE]

holds for each $k$. Also, by the second condition of $\mathcal{A}$,

[TABLE]

holds. Thus, if we define $\widetilde{a}:=(a_{1},\ldots,a_{\lceil k^{2\alpha/\beta}\rceil},0,0,\ldots)$ and $\widetilde{i}_{1},\widetilde{i}_{2},\ldots$ similarly,

[TABLE]

satisfies $\|a-b\|_{\ell^{2}}\leq\sqrt{C_{1}/2\alpha+C_{2}}\cdot k^{-\alpha}$. Then its quantization

[TABLE]

satisfies $\|b-\widetilde{b}\|_{\ell^{2}}\leq k^{-\alpha}$ as $b$ has at most $k$ nonzero terms. Since $|b_{i}|\leq C_{1}$, the number of values possibly taken by $\widetilde{b}_{i}$ is at most $2C_{1}k^{1/2+\alpha}+1$. Hence, the logarithm of the number of such $\widetilde{b}$ values can be upper-bounded by

[TABLE]

where $C_{0}>0$ is a constant. Since $\|a-\widetilde{b}\|_{\ell^{2}}\leq(1+\sqrt{C_{1}/2\alpha+C_{2}})k^{-\alpha}$ holds, if we take $k\sim\varepsilon^{-1/\alpha}$ in the same way as used in the proof of lower-bound, then we reach the conclusion.

Remark B.4**.**

In the case in which $\beta\geq 2\alpha$ holds, we can obtain a more accurate bound by using Stirling’s approximation. However, we can see that such a case no longer requires the concept of weak $\ell^{p}$ norms, or else its conditions are too strong. Therefore, we have not treated this case.

B.6 Proof of Theorem 4.15

This proof is a refinement of the proof of Theorem 1 in Zhang et al. (2002). First, we prove the following lemma.

Lemma B.5**.**

Let $\gamma\in(0,1)$, and let $m$ be a positive integer such that $m\leq n^{\gamma}\leq 2m$. For $k=0,\ldots,m-1$, let $A_{k}$ be the number of $X_{1},\ldots,X_{n}$ contained in $[k/m,(k+1)/m)$. Then there exists a constant $c=c(\gamma)>0$ such that

[TABLE]

holds for any $m,n$ satisfying the condition.

Proof.

For a fixed $k$, $A_{k}$ can be written as $A_{k}=\sum_{j=1}^{n}\eta_{j}$, where $(\eta_{j})_{j=1}^{n}$ is an i.i.d. sequence with $\mathrm{P}(\eta_{j}=0)=1/m$ and $\mathrm{P}(\eta_{j}=1)=1-1/m$. Then, by Chernoff’s inequality, we have for $t>0$

[TABLE]

Setting $t=\log 2$ and assuming $c>\log 2$, we obtain

[TABLE]

where we have used the fact that $(1+1/x)^{x}$ is increasing on $x>0$. Then we finally have

[TABLE]

Considering the logarithm of the last term, it is sufficient to take $c$ as large enough to satisfy

[TABLE]

and we obtain the conclusion. ∎

We now prove Theorem 4.15. Let $\gamma=\frac{1}{1+\beta}$, and let

[TABLE]

Fix a linear estimator $\widehat{f}(x)=\sum_{i=1}^{n}Y_{i}\varphi_{i}(x;X^{n})$. For any $f^{\circ}\in BV(C)$, we have by Fubini’s theorem

[TABLE]

Take a sufficiently large $n$, and let $m$ be a power of $2$ in $[\frac{1}{2}n^{\gamma},n^{\gamma}]$. Notice that it holds that $m\leq n^{\gamma}\leq 2m$. Then there exists an integer $0\leq k<m$ such that

[TABLE]

Let $f^{\circ}=Fm^{-\beta/2}\cdot m^{1/2}\psi(m\cdot-k)$, where $F:=\min\{C_{1},C_{2}^{1/2}\}$. Then we have $f^{\circ}\in\mathcal{J}^{p}_{\psi}(C_{1},C_{2},\beta)$. Let $A$ denote an event assured to have a probability of at least $1-2^{-n^{1-\gamma}}$ in Lemma B.5, and we obtain

[TABLE]

where $M$ is the number of sections $[\ell,\ell+1)$ such that $\ell$ is an integer and $[\ell,\ell+1)\cap\mathrm{supp}(\psi)$ is not empty. The last inequality has been derived by (30). By (29), (31), and the triangle inequality,

[TABLE]

Since $n$ is sufficiently large, we can assume $1-2^{-n^{1-\gamma}}\geq 1/2$. Define a constant $G$ by

[TABLE]

We have $m^{1+\beta}=m^{1/\gamma}\geq(\frac{1}{2}n^{\gamma})^{1/\gamma}=2^{-1/\gamma}$ by assumption, and so we obtain

[TABLE]

as desired.

B.7 Proof of Lemma 5.2

(mainly following Suzuki (2019)) For $f\in\mathcal{N}(L,S,D,B)$ expressed as

[TABLE]

let us define

[TABLE]

for $k=1,\ldots,L+1$, where $\mathcal{A}_{1}(f)$ denotes the identity map, and $\mathcal{B}_{L+1}(f)=W_{L+1}$. Then $f=\mathcal{B}_{k+1}(f)\circ\rho(W_{k}\cdot-v_{k})\circ\mathcal{A}_{k}(f)$ holds for $k=1,\ldots,L$. Here, notice that for each $x\in[0,1]^{d}$ and $1\leq k\leq L+1$,

[TABLE]

holds, where we have used the assumption $B\geq 1$ at the last inequality. Also, the Lipschitz continuity of $\mathcal{B}_{k}(f)$ can be derived as

[TABLE]

Let $\varepsilon>0$. Suppose $f,g\in\mathcal{N}(L,S,D,B)$ satisfy

[TABLE]

and $\|W_{i}-W^{\prime}_{i}\|_{\infty}\leq\varepsilon$, $\|v_{i}-v_{i}^{\prime}\|_{\infty}\leq\varepsilon$ for each $i$. Then we have by (32) and (33)

[TABLE]

Therefore, for a fixed sparsity pattern, and letting $\varepsilon=\left((L+1)B^{L}(D+1)^{L+1}\right)^{-1}\delta$, the $\delta$-covering number is bounded by

[TABLE]

The number of such patterns is bounded by $\binom{(D+1)^{L}}{S}\leq(D+1)^{LS}$, and so the $\delta$-covering entropy is bounded by

[TABLE]

as desired.

B.8 Proof of Theorem 5.10

Let $N$ be an integer in $[n^{\frac{1}{2\alpha+1}},2n^{\frac{1}{2\alpha+1}}]$. Also, suppose that we have an integer $m$ in $[\frac{2\alpha}{\beta(2\alpha+1)}\log_{2}n,$ $\frac{4\alpha}{\beta(2\alpha+1)}\log_{2}n]$ (for sufficiently large $n$). Fix the target function

[TABLE]

Then, let $(k^{1},\ell^{1}),\ldots,(k^{N},\ell^{N})\in T_{0}\setminus T_{m}$ be the $N$ (absolutely) largest coefficients $a_{k^{1},\ell^{1}},\ldots,a_{k^{N},\ell^{N}}$. Let

[TABLE]

Then we have

[TABLE]

Next, we approximate $\sum_{(k,\ell)\in T}a_{k,\ell}\psi_{k,\ell}$ by some neural network. Now, $(k,\ell)\in T$ implies that

[TABLE]

satisfies $\max_{1\leq i\leq d}k_{i}<m$. Let $\widetilde{\psi}\in\mathcal{N}(L_{\varepsilon},S_{\varepsilon},D_{\varepsilon},B_{\varepsilon})$ satisfy $\|\widetilde{\psi}-\psi\|_{L^{2}}\leq\varepsilon$. Since we have

[TABLE]

we can construct the approximator

[TABLE]

in a manner similar to that shown in Fig. 4 such that

[TABLE]

holds. If we determine $\varepsilon=1/n$ and define $\mathcal{N}_{F}$ by using the set defined in (37), we have, by Lemma 5.2 and the assumption,

[TABLE]

for some constant $C^{\prime}>0$.

Combining (36), (37), (39), and Theorem 2.6, we obtain an evaluation

[TABLE]

for some $C>0$, where $\widehat{f}$ denotes the empirical risk minimizer over $\mathcal{N}_{F}$.

B.9 Proof of Theorem 5.14

We consider functions expressed as $f^{N}=\sum_{i=1}^{N}c_{i}f(A_{i}\cdot-b_{i})\in\mathcal{N}^{N}(L,S,D,B)$, where $f\in\mathcal{N}(L,S,D,B)$. Then it suffices to consider $f$ defined over $[-B(d+1),B(d+1)]^{d}\subset[-B(D+1),B(D+1)]^{d}$. This changes evaluation (32) to $\|\mathcal{A}_{k}(f)(x)\|_{\infty}\leq B^{k}(D+1)^{k}$, and so $B^{L+1}(D+1)^{L+1}$ in (34) is replaced by $B^{L+2}(D+1)^{L+2}$ in the evaluation of the covering entropy with the domain limited to $[-B(D+1),B(D+1)]^{d}$. However, the upper bound (35) is still valid if $L\geq 2$, and so we have a set $\mathcal{N}_{\varepsilon}$ for each $0<\varepsilon<1$ satisfying

• •

$\mathcal{N}_{\varepsilon}$ is an $\varepsilon$-covering of $\mathcal{N}(L,S,D,B)$ with respect to $\|\cdot\|_{L^{\infty}}$, where the domain is limited to $[-B(D+1),B(D+1)]^{d}$;

• •

$\log|\mathcal{N}_{\varepsilon}|\leq\displaystyle 2S(L+1)\log\left(\frac{B(L+1)(D+1)}{\varepsilon}\right)$.

For $x\in[0,1]^{d}$, we have $|f(A_{i}x-b_{i})|\leq B^{L+2}(D+1)^{L+2}$ by an evaluation similar to the one in (32). Also, we have the Lipschitz continuity $|f(x)-f(x^{\prime})|\leq B^{L+1}D^{L+1}\|x-x^{\prime}\|_{\infty}$, similar to (33).

Let us consider two functions in $\mathcal{N}^{N}(L,S,D,B)$ expressed as

[TABLE]

such that $\|f-g\|_{L^{\infty}([-B(D+1),B(D+1)]^{d})}\leq\varepsilon$ holds and $\|A_{i}-A_{i}^{\prime}\|_{\infty},\|b_{i}-b_{i}^{\prime}\|_{\infty},|c_{i}-c_{i}^{\prime}|\leq\varepsilon$ holds for each $i$. Then we have for each $x\in[0,1]^{d}$

[TABLE]

If we determine $\varepsilon=(3NB^{L+2}(D+1)^{L+2})^{-1}\delta$ for $0<\delta<1$, the $\delta$-covering entropy of $\mathcal{N}^{N}(L,S,D,B)$ is now bounded by

[TABLE]

as desired.

B.10 Proof of Lemma A.1

The probability law $\mathrm{P}_{f}$ generated by $f$ is regarded as being on $\mathbb{R}^{d}\times\mathbb{R}$. Hence, its density at $z=(x,y)$ is

[TABLE]

As $p_{g}$ can be calculated in the same way, we have

[TABLE]

This coincides with the expectation of \frac{1}{2\sigma^{2}}\bigl{(}(Y-g(X))^{2}-(Y-f(X))^{2}\bigr{)} with $Y=f(X)+\xi$. The term in parentheses is calculated as

[TABLE]

where we have used the facts that each $X$ follows the uniform distribution over $[0,1]^{d}$ and that each $\xi$ is independent of $X$. Thus, we obtain the desired result.