Function values are enough for $L_2$-approximation

TL;DR

This paper demonstrates that function values are nearly as effective as arbitrary linear information for $L_2$-approximation in Hilbert spaces, establishing decay rates of sampling numbers and challenging existing assumptions about optimal algorithms.

Contribution

It proves that sampling numbers decay at the same rate as approximation numbers for certain function spaces, showing function values are nearly as powerful as arbitrary linear information.

Findings

Sampling numbers decay with the same polynomial rate as approximation numbers.

Function values are essentially as powerful as arbitrary linear information for $L_2$-approximation.

Improves bounds for Sobolev spaces with dominating mixed smoothness and disproves the optimality of Smolyak's algorithm.

Abstract

We study the $L_2$-approximation of functions from a Hilbert space and compare the sampling numbers with the approximation numbers. The sampling number $e_n$ is the minimal worst case error that can be achieved with $n$ function values, whereas the approximation number $a_n$ is the minimal worst case error that can be achieved with $n$ pieces of arbitrary linear information (like derivatives or Fourier coefficients). We show that \[ e_n \,\lesssim\, \sqrt{\frac{1}{k_n} \sum_{j\geq k_n} a_j^2}, \] where $k_n \asymp n/\log(n)$. This proves that the sampling numbers decay with the same polynomial rate as the approximation numbers and therefore that function values are basically as powerful as arbitrary linear information if the approximation numbers are square-summable. Our result applies, in particular, to Sobolev spaces $H^s_{\rm mix}(\mathbb{T}^d)$ with dominating mixed smoothness $s>1/2$ and we obtain \[ e_n \,\lesssim\, n^{-s} \log^{sd}(n). \] For $d>2s+1$, this improves upon all previous bounds and disproves the prevalent conjecture that Smolyak's (sparse grid) algorithm is optimal.

Equations

$$e_n\lesssim \sqrt{\frac{1}{k_n}\sum _{j\ge k_n}{a}_j^2},$$

$$e_n\lesssim n^{-s}{\log }^{sd}(n).$$

$${\delta }_x:H\to \mathbb{R},{\delta }_x(f):=f(x)$$

$$e_{n}\,:=\,e_{n}(H)\,:=\,\inf_{\begin{subarray}{c}x_{1},\dots,x_{n}\in D\\ \varphi_{1},\dots,\varphi_{n}\in L_{2}\end{subarray}}\,\sup_{f\in H\colon\|f\|_{H}\leq 1}\,\Big{\|}f-\sum_{i=1}^{n}f(x_{i})\,\varphi_{i}\Big{\|}_{L_{2}},$$

$$a_{n}\,:=\,a_{n}(H)\,:=\,\inf_{\begin{subarray}{c}L_{1},\dots,L_{n}\in H^{\prime}\\ \varphi_{1},\dots,\varphi_{n}\in L_{2}\end{subarray}}\,\sup_{f\in H\colon\|f\|_{H}\leq 1}\,\Big{\|}f-\sum_{i=1}^{n}L_{i}(f)\,\varphi_{i}\Big{\|}_{L_{2}},$$

$$\underset{n\to \infty }{\lim }{e}_n=0\iff \underset{n\to \infty }{\lim }{a}_n=0\iff H\hookrightarrow L_2\text{ compactly},$$

$$\mathrm{ord}(c_n):=\sup \left\{s\ge 0:\underset{n\to \infty }{\lim }{c}_n{n}^s=0\right\}$$

$$\mathrm{ord}(e_n)\ge \frac{2\mathrm{ord}(a_n)}{2\mathrm{ord}(a_n)+1}\mathrm{ord}(a_n).$$

$$e_n(H{)}^2\le \frac{C}{k_n}\sum _{j\ge k_n}{a}_j(H{)}^2.$$

$$e_n(H)\lesssim n^{-s}{\log }^{\alpha +s}(n).$$

$$H:=H^{s}_{\rm mix}(\mathbb{T}^{d}):=\Big{\{}f\in L_{2}(\mathbb{T}^{d})\,\Big{|}\,\|f\|_{H}^{2}:=\sum_{{\bf m}\in\mathbb{N}_{0}^{d}}\prod_{j=1}^{d}(1+|{m_{j}}|^{2s})\left\langle f,\bf b_{{\bf m}}\right\rangle_{L_{2}}^{2}<\infty\Big{\}}.$$

$$\Vert f{\Vert }_{H_{\mathrm{mix}}^s({\mathbb{T}}^d)}^2=\sum _{\alpha \in \{0,s{\}}^d}\Vert D^{\alpha }f{\Vert }_{L_2}^2.$$

$$a_n\asymp n^{-s}{\log }^{s(d-1)}(n)$$

$$n^{-s}{\log }^{s(d-1)}(n)\lesssim e_n\lesssim n^{-s}{\log }^{(s+1/2)(d-1)}(n)$$

$$e_{n}\big{(}H^{s}_{\rm mix}(\mathbb{T}^{d})\big{)}\,\lesssim\,n^{-s}\log^{sd}(n).$$

$$f=\sum _{j=1}^{\infty }{f}_j{b}_j,\text{ where }{f}_j:={⟨f,b_j⟩}_{L_2},$$

$$P_n:H\to L_2,P_n(f):=\sum _{j\le n}{f}_j{b}_j,$$

$$a_{n}(H)=\sup_{f\in H\colon\|f\|_{H}\leq 1}\big{\|}f-P_{n}(f)\big{\|}_{L_{2}}=\|b_{n+1}\|_{H}^{-1}.$$

$$\varrho :D\to \mathbb{R},\varrho (x):=\frac{1}{2}\left(\frac{1}{k}\sum _{j<k}{b}_{j+1}(x{)}^2+\frac{1}{{\sum }_{j\ge k}{a}_j^2}\sum _{j\ge k}{a}_j^2{b}_{j+1}(x{)}^2\right),$$

$$A_n:H\to L_2,A_n(f):=\sum _{j=1}^k(G^{+}Nf{)}_j{b}_j,$$

$$G:=(\varrho (x_i{)}^{-1/2}{b}_j(x_i){)}_{i\le n,j\le k}\in {\mathbb{R}}^{n\times k}.$$

$$A_n(f)=\underset{g\in V_k}{\mathrm{argmin}}\sum _{i=1}^n\frac{|g(x_i)-f(x_i){|}^2}{\varrho (x_i)}.$$

$$e(A_{n}):=\sup_{f\in H\colon\|f\|_{H}\leq 1}\,\big{\|}f-A_{n}(f)\big{\|}_{L_{2}}.$$

$$e^{\mathrm{ran}}(A_n):=\underset{f\in H:\Vert f{\Vert }_H\le 1}{\sup }{\left(\mathbb{E}{\Vert f-A_n(f)\Vert }_{L_2}^2\right)}^{1/2}.$$

$$e^{\mathrm{ran}}(M_n)\lesssim n^{-s}{\log }^{\alpha }(n).$$

$$\begin{array}{rl}{\Vert f-A_{n}f\Vert }_{L_2}& \le a_k+{\Vert P_{k}f-A_{n}f\Vert }_{L_2}=a_k+{\Vert A_n(f-P_{k}f)\Vert }_{L_2}\\ & =a_k+{\Vert G^{+}N(f-P_{k}f)\Vert }_{{\ell }_2^k}\\ & \le a_k+\Vert G^{+}:{\ell }_2^n\to {\ell }_2^k\Vert \Vert N:P_k(H{)}^{\perp }\to {\ell }_2^n\Vert .\end{array}$$

$$\Gamma:=\big{(}\varrho(x_{i})^{-1/2}a_{j}b_{j+1}(x_{i})\big{)}_{1\leq i\leq n,j\geq k}\in\mathbb{R}^{n\times\infty}.$$

$$e(A_n)\le a_k+\frac{s_{\max }(\Gamma )}{s_{\min }(G)}.$$

$$\beta_{k}\,:=\,\left(\frac{1}{k}\sum_{j\geq k}a_{j}^{2}\right)^{1/2}\qquad\text{ and }\qquad\gamma_{k}\,:=\,\max\Big{\{}a_{k},\,\beta_{k}\Big{\}}.$$

$${\beta }_k\asymp \{\begin{array}{ll}{a}_k,& \text{if }s>1/2,\\ a_k\sqrt{\log (k)},& \text{if }s=1/2\text{ and }\alpha <-1/2,\end{array}$$

Full text

[TABLE]

where $k_{n}\asymp n/\log(n)$. This proves that the sampling numbers decay with the same polynomial rate as the approximation numbers and therefore that function values are basically as powerful as arbitrary linear information if the approximation numbers are square-summable. Our result applies, in particular, to Sobolev spaces $H^{s}_{\rm mix}(\mathbb{T}^{d})$ with dominating mixed smoothness $s>1/2$ and dimension $d\in\mathbb{N}$ and we obtain

[TABLE]

For $d>2s+1$, this improves upon all previous bounds and disproves the prevalent conjecture that Smolyak’s (sparse grid) algorithm is optimal.

D. Krieg is supported by the Austrian Science Fund (FWF) Project F5513-N26, which is a part of the Special Research Program Quasi-Monte Carlo Methods: Theory and Applications.

Let $H$ be a reproducing kernel Hilbert space, i.e., a Hilbert space of real-valued functions on a set $D$ such that point evaluation

[TABLE]

is a continuous functional for all $x\in D$. We consider numerical approximation of functions from such spaces, using only function values. We measure the error in the space $L_{2}=L_{2}(D,\mathcal{A},\mu)$ of square-integrable functions with respect to an arbitrary measure $\mu$ such that $H$ is embedded into $L_{2}$. This means that the functions in $H$ are square-integrable and two functions from $H$ that are equal $\mu$-almost everywhere are also equal point-wise.

We are interested in the $n$-th minimal worst-case error

[TABLE]

which is the worst-case error of an optimal algorithm that uses at most $n$ function values. These numbers are sometimes called sampling numbers. We want to compare $e_{n}$ with the $n$-th approximation number

[TABLE]

where $H^{\prime}$ is the space of all bounded, linear functionals on $H$. This is the worst-case error of an optimal algorithm that uses at most $n$ linear functionals as information. Clearly, we have $a_{n}\leq e_{n}$ since the point evaluations form a subset of $H^{\prime}$.

The approximation numbers are quite well understood in many cases because they are equal to the singular values of the embedding operator ${\rm id}\colon H\to L_{2}$. However, the sampling numbers still resist a precise analysis. For an exposition of such approximation problems we refer to [11, 12, 13], especially [13, Chapter 26 & 29], and references therein. One of the fundamental questions in the area asks for the relation of $e_{n}$ and $a_{n}$ for specific Hilbert spaces $H$. The minimal assumption on $H$ is the compactness of the embedding ${\rm id}\colon H\to L_{2}$. It is known that

[TABLE]

see [13, Section 26.2]. However, the compactness of the embedding is not enough for a reasonable comparison of the speed of this convergence, see [6]. If $(a_{n}^{*})$ and $(e_{n}^{*})$ are decreasing sequences that converge to zero and $(a_{n}^{*})\not\in\ell_{2}$, one may construct $H$ and $L_{2}$ such that $a_{n}=a_{n}^{*}$ for all $n\in\mathbb{N}$ and $e_{n}\geq e_{n}^{*}$ for infinitely many $n\in\mathbb{N}$. In particular, if

[TABLE]

denotes the (polynomial) order of convergence of a positive sequence $(c_{n})$, it may happen that $\operatorname{ord}(e_{n})=0$ even if $\operatorname{ord}(a_{n})=1/2$.

It thus seems necessary to assume that $(a_{n})$ is in $\ell_{2}$, i.e., that ${\rm id}\colon H\to L_{2}$ is a Hilbert-Schmidt operator. This is fulfilled, e.g., for Sobolev spaces defined on the unit cube, see Corollary 3. Under this assumption, it is proven in [9] that

[TABLE]

In fact, the authors of [9] conjecture that the order of convergence is the same for both sequences. We give an affirmative answer to this question. Our main result can be stated as follows.

[TABLE]

In particular, we obtain the following result on the order of convergence. This solves Open Problem 126 in [13, p. 333], see also [13, Open Problems 140 & 141].

[TABLE]

In particular, we always have $\operatorname{ord}(e_{n})=\operatorname{ord}(a_{n})$.

Let us now consider a specific example. Namely, we consider Sobolev spaces with (dominating) mixed smoothness defined on the $d$-dimensional torus $\mathbb{T}^{d}\cong[0,1)^{d}$. These spaces attracted quite a lot of attention in various areas of mathematics due to their intriguing attributes in high-dimensions. For history and the state of the art (from a numerical analysis point of view) see [3, 19, 20].

Let us first define a one-dimensional and real-valued orthonormal basis of $L_{2}(\mathbb{T})$ by $b_{0}^{(1)}=1$, $b_{2m}^{(1)}=\sqrt{2}\cos(2\pi mx)$ and $b_{2m-1}^{(1)}=\sqrt{2}\sin(2\pi mx)$ for $m\in\mathbb{N}$. From this we define a basis of $L_{2}(\mathbb{T}^{d})$ using $d$-fold tensor products: We set $\mathbf{b}_{\bf m}:=\bigotimes_{j=1}^{d}b_{m_{j}}^{(1)}$ for ${\bf m}=(m_{1},\dots,m_{d})\in\mathbb{N}_{0}^{d}$. The Sobolev space with dominating mixed smoothness $s>0$ can be defined as

[TABLE]

This is a Hilbert space. It satisfies our assumptions whenever $s>1/2$. It is not hard to prove that an equivalent norm in $H^{s}_{\rm mix}(\mathbb{T}^{d})$ for $s\in\mathbb{N}$ is given by

[TABLE]

The approximation numbers $a_{n}=a_{n}(H)$ are known for some time to satisfy

[TABLE]

for all $s>0$, see e.g. [3, Theorem 4.13]. The sampling numbers $e_{n}=e_{n}(H)$, however, seem to be harder to tackle. The best bounds so far are

[TABLE]

for $s>1/2$. The lower bound easily follows from $e_{n}\geq a_{n}$, and the upper bound was proven in [17], see also [3, Chapter 5]. For earlier results on this prominent problem, see [15, 16, 18, 22]. Note that finding the right order of $e_{n}$ in this case is posed as Outstanding Open Problem 1.4 in [3]. From Corollary 2, setting $\alpha=s(d-1)$ in the second part, we easily obtain the following.

[TABLE]

The bound in Corollary 3 improves on the previous bounds if $d>2s+1$, or equivalently $s<(d-1)/2$. With this, we disprove Conjecture 5.26 from [3] and show, in particular, that Smolyak’s algorithm is not optimal in these cases. Although our techniques do not lead to an explicit deterministic algorithm that achieves the above bounds, it is interesting that $n$ i.i.d. random points are suitable with positive probability (independent of $n$).

Let us conclude with a few topics for future research. While this paper was under review, Theorem 1 has already been extended to the case of complex-valued functions and non-injective operators ${\rm id}\colon H\to L_{2}$ in [7], including explicit values for the constants $c$ and $C$, see also [21]. It remains open to generalize our results to non-Hilbert space settings. It is also quite a different question whether the sampling numbers and the approximation numbers behave similarly with respect to the dimension of the domain $D$. This is a subject of tractability studies. We refer to [13, Chapter 26] and especially [14, Corollary 8]. Here, we only note that the constants of Theorem 1 are, in particular, independent of the domain, and that this may be utilized for these studies, see also [7].

The Proof

The result follows from a combination of the general technique to assess the quality of random information as developed in [4, 5], together with bounds on the singular values of random matrices with independent rows from [10].

Before we consider algorithms that only use function values, let us briefly recall the situation for arbitrary linear functionals. In this case, the minimal worst-case error $a_{n}$ is given via the singular value decomposition of ${\rm id}:H\to L_{2}$ in the following way. Since $W={\rm id}^{*}{\rm id}$ is positive, compact and injective, there is an orthogonal basis $\mathcal{B}=\left\{b_{j}\colon j\in\mathbb{N}\right\}$ of $H$ that consists of eigenfunctions of $W$. Without loss of generality, we may assume that $H$ is infinite-dimensional. It is easy to verify that $\mathcal{B}$ is also orthogonal in $L_{2}$. We may assume that the eigenfunctions are normalized in $L_{2}$ and that $\|b_{1}\|_{H}\leq\|b_{2}\|_{H}\leq\dots$. From these properties, it is clear that the Fourier series

[TABLE]

converges in $H$ for every $f\in H$, and therefore also point-wise. The optimal algorithm based on $n$ linear functionals is given by

[TABLE]

which is the $L_{2}$-orthogonal projection onto $V_{n}:={\rm span}\{b_{1},\ldots,b_{n}\}$. We refer to [11, Section 4.2] for details. We obtain that

[TABLE]

We now turn to algorithms using only function values. In order to bound the minimal worst-case error $e_{n}$ from above, we employ the probabilistic method in the following way. Let $x_{1},\dots,x_{n}\in D$ be i.i.d. random variables with $\mu$-density

[TABLE]

where $k\leq n$ will be specified later. Given these sampling points, we consider the algorithm

[TABLE]

where $N\colon H\to\mathbb{R}^{n}$ with $N(f):=(\varrho(x_{i})^{-1/2}f(x_{i}))_{i\leq n}$ is the weighted information mapping and $G^{+}\in\mathbb{R}^{k\times n}$ is the Moore-Penrose inverse of the matrix

[TABLE]

This algorithm is a weighted least squares estimator: If $G$ has full rank, then

[TABLE]

In particular, we have $A_{n}(f)=f$ whenever $f\in V_{k}$. The worst-case error of $A_{n}$ is defined as

[TABLE]

Clearly, we have $e_{n}\leq e(A_{n})$ for every realization of $x_{1},\dots,x_{n}$. Thus, it is enough to show that $e(A_{n})$ obeys the desired upper bound with positive probability.

Remark 1**.**

If $\mu$ is a probability measure and if the basis is uniformly bounded, i.e., if $\sup_{j\in\mathbb{N}}\,\|b_{j}\|_{\infty}<\infty$, we may also choose $\varrho\equiv 1$ and consider i.i.d. sampling points with distribution $\mu$.

Remark 2**.**

Weighted least squares estimators are widely studied in the literature. We refer to [1, 2]. In contrast to previous work, we show that we can choose a fixed set of weights and sampling points that work simultaneously for all $f\in H$. We do not need additional assumptions on the function $f$, the basis $(b_{j})$ or the measure $\mu$. For this, we think that our modification of the weights is important.

Remark 3**.**

The worst-case error $e(A_{n})$ of the randomly chosen algorithm $A_{n}$ is not to be confused with the Monte Carlo error of a randomized algorithm, which can be defined by

[TABLE]

The Monte-Carlo error is a weaker error criterion. It is shown in [8], see also [23], that the assumptions of Corollary 2 give rise to a randomized algorithm $M_{n}$ which uses at most $n$ function values and satisfies

[TABLE]

However, this does not imply that the worst-case error $e(M_{n})$ is small for any realization of $M_{n}$.

To give an upper bound on $e(A_{n})$, let us assume that $G$ has full rank. For any $f\in H$ with $\|f\|_{H}\leq 1$, we have

[TABLE]

The norm of $G^{+}$ is the inverse of the $k$th largest (and therefore the smallest) singular value of the matrix $G$. The norm of $N$ is the largest singular value of the matrix

[TABLE]

To see this, note that $N=\Gamma\Delta$ on $P_{k}(H)^{\perp}$, where the mapping $\Delta\colon P_{k}(H)^{\perp}\to\ell_{2}$ with $\Delta g=(g_{j+1}/a_{j})_{j\geq k}$ is an isomorphism. This yields

[TABLE]

It remains to bound $s_{\rm min}(G)$ from below and $s_{\rm max}(\Gamma)$ from above. Clearly, any nontrivial lower bound on $s_{\rm min}(G)$ automatically yields that the matrix $G$ has full rank. To state our results, let

[TABLE]

Note that $a_{2k}^{2}\leq\frac{1}{k}(a_{k}^{2}+\ldots+a_{2k}^{2})\leq\beta_{k}^{2}$  for all $k\in\mathbb{N}$ and thus $\gamma_{k}\leq\beta_{\lfloor k/2\rfloor}$. Before we continue with the proof of Theorem 1, we show that Corollary 2 follows from Theorem 1 by providing the order of $\beta_{k}$ in the following special case. The proof is an easy exercise.

Lemma 1**.**

Let $a_{n}\asymp n^{-s}\log^{\alpha}(n)$ for some $s,\alpha\in\mathbb{R}$. Then,

[TABLE]

and $\beta_{k}=\infty$ in all other cases.

The rest of the paper is devoted to the proof of the following two claims: There exist constants $c,C>0$ such that, for all $n\in\mathbb{N}$ and $k=\lfloor c\,n/\log n\rfloor$, we have

Claim 1:

[TABLE]

Claim 2:

[TABLE]

Together with (1), this will yield with positive probability that

[TABLE]

which is the statement of Theorem 1.

Both claims are based on [10, Theorem 2.1], which we state here in a special case. Recall that, for $X\in\ell_{2}$, the operator $X\otimes X$ is defined on $\ell_{2}$ by $X\otimes X(v)=\langle X,v\rangle_{2}\cdot X$. By $\left\|M\right\|$ we denote the spectral norm of a matrix $M$.

Proposition 1**.**

There exists an absolute constant $c>0$ for which the following holds. Let $X$ be a random vector in $\mathbb{R}^{k}$ or $\ell_{2}$ with $\|X\|_{2}\leq R$ with probability 1, and let $X_{1},X_{2},\dots$ be independent copies of $X$. We put

[TABLE]

Then, for any $t>0$,

[TABLE]

Proof of Proposition 1.

We describe the steps needed to obtain this reformulation of [10, Theorem 2.1]. For this let $\|Z\|_{\psi_{\alpha}}:=\inf\{C>0\colon\mathbb{E}\exp(|Z|^{\alpha}/C^{\alpha})\leq 2\}$ for $Z=\|X\|_{2}$ and

[TABLE]

Theorem 2.1 of [10] then states that

[TABLE]

with

[TABLE]

for all $t>0$, $\alpha\geq 1$, $n\in\mathbb{N}$ and some absolute constant $c>0$. Note that the 2 in the right hand side of above inequality is missing in [10, Theorem 2.1], but can be found in the proof.

From $\|X\|_{2}\leq R$ we obtain $\|Z\|_{\psi_{\alpha}}\leq 2R$ for all $\alpha\geq 1$. Therefore, we can take the limit $\alpha\to\infty$ and obtain the result with $\widetilde{A}=\frac{R^{2}\log n}{n}$ and $\widetilde{B}=\frac{\rho^{2}}{\sqrt{n}}+R\|D\|^{1/2}\sqrt{\frac{\log n}{n}}$ (and a slightly changed constant $c$). Moreover, we have

[TABLE]

for any $\theta\in\mathbb{R}^{k}$ (or $\ell_{2}$) with $\|\theta\|_{2}=1$, which implies $\rho^{2}\leq R\cdot\|D\|^{1/2}$. This “trick” leads to an improvement over [10, Corollary 2.6] and yields our formulation of the result.

∎

Proof of Claim 1.

Consider independent copies $X_{1},\ldots,X_{n}$ of the vector

[TABLE]

where $x$ is a random variable on $D$ with density $\varrho$. Clearly, $\sum_{i=1}^{n}X_{i}\otimes X_{i}=\Gamma^{*}\Gamma$ with $\Gamma$ from above. First observe

[TABLE]

Since $D:=\mathbb{E}(X\otimes X)=\mathop{\mathrm{diag}}(a_{k}^{2},a_{k+1}^{2},\ldots)$ we have $\|D\|=a_{k}^{2}$. This implies, with $A$ and $B$ defined as in Proposition 1, that

[TABLE]

and

[TABLE]

Choosing $k=\lfloor c\,n/\log n\rfloor$ for $c$ small enough, we obtain

[TABLE]

By choosing $t=2$, we obtain with probability greater $1/2$ that

[TABLE]

This yields Claim 1.

∎

Proof of Claim 2.

Consider $X:=\varrho(x)^{-1/2}(b_{1}(x),\ldots,b_{k}(x))$ with $x$ distributed according to $\varrho$. Clearly, $\sum_{i=1}^{n}X_{i}\otimes X_{i}=G^{*}G$ with $G$ from above. First observe

[TABLE]

Since $D:=\mathbb{E}(X\otimes X)=\mathop{\mathrm{diag}}(1,\ldots,1)$ we have $\|D\|=1$. This implies, with $A$ and $B$ defined as in Proposition 1, that

[TABLE]

and

[TABLE]

Again, choosing $k=\lfloor c\,n/\log n\rfloor$ for $c$ small enough, we obtain

[TABLE]

By choosing $t=2$, we obtain with probability greater $1/2$ that

[TABLE]

This yields Claim 2.

∎