On the power of random information

TL;DR

This paper investigates how random information compares to optimal information in approximation and integration problems, showing that randomness can be nearly optimal in some cases and significantly worse in others.

Contribution

It provides new theoretical results and a survey on the effectiveness of random versus optimal information in computational problems.

Findings

Random information is nearly optimal for some problems.

In other problems, random information performs much worse than optimal.

The paper offers a comprehensive overview of known and new results.

Abstract

We study approximation and integration problems and compare the quality of optimal information with the quality of random information. For some problems random information is almost optimal and for some other problems random information is much worse than optimal information. We prove new results and give a short survey of known results.

Equations

$$N_{n}:F\to\mathbb{R}^{n},\quad N_{n}(f)=\big{(}L_{1}(f),\ldots,L_{n}(f)\big{)},$$

$$r(N_n)=\underset{\phi :{\mathbb{R}}^n\to G}{\inf }\underset{f\in F}{\sup }{\Vert S(f)-\phi (N_n(f))\Vert }_G.$$

$$r(N_n^{\ast })=\underset{N_n}{\inf }r(N_n).$$

$$r(N_n^{\ast })\text{vs.}\mathbb{E}[r(N_n)],$$

$$\mathrm{APP}:W_p^1([0,1])\to L_q([0,1])$$

$$\Vert f{\Vert }_{W_p^1([0,1])}:={\left(\Vert f{\Vert }_p^p+\Vert f^{\prime }{\Vert }_p^p\right)}^{1/p}.$$

$$N_{n}^{\ast}(f)=\big{(}f(1/n),f(3/n),\dots,f((2n-1)/n)\big{)},$$

$$r(N_n^{\ast })\asymp n^{-1+(1/p-1/q{)}_{+}},$$

$$\mathbb{E}[r(N_n)]\asymp \{\begin{array}{ll}r(N_n^{\ast })\asymp n^{-1}& \text{if}p>q\\ & \\ r(N_{n/\log n}^{\ast })\asymp {\left(\frac{n}{\log n}\right)}^{-1+1/p-1/q}& \text{if}p\le q.\end{array}$$

$$r(N_{n})\asymp\left\{\begin{array}[]{ll}\displaystyle\left(\sum_{i=0}^{n}(x_{i+1}-x_{i})^{(pq+p-q)/(p-q)}\right)^{1/q-1/p}&\text{if}\quad p>q\\ \displaystyle\max_{0\leq i\leq n}(x_{i+1}-x_{i})^{1-1/p+1/q}&\text{if}\quad p\leq q.\\ \end{array}\right.$$

$$r(N_n)\asymp \underset{\begin{array}{c}f\in W_p^1([0,1])\\ N_n(f)=0\end{array}}{\sup }\frac{{\Vert f\Vert }_q}{\Vert f{\Vert }_{W_p^1}},$$

$$\ell =x_{i_0+1}-x_{i_0}=\underset{0\le i\le n}{\max }(x_{i+1}-x_i)$$

$$N_n(f)=(0,\dots ,0),\Vert f{\Vert }_q\asymp {\ell }^{1+1/q},\text{and}\Vert f{\Vert }_{W_p^1}\asymp {\ell }^{1/p}$$

$$\Vert f{\Vert }_q\asymp {\left(\sum _{i=0}^n{\ell }_i^{(pq+p-q)/(p-q)}\right)}^{1/q}\text{and}\Vert f{\Vert }_{W_p^1}\asymp {\left(\sum _{i=0}^n{\ell }_i^{(pq+p-q)/(p-q)}\right)}^{1/p}$$

$$\Vert f{\Vert }_q={\left(\sum _{i=0}^n\Vert f_i{\Vert }_q^q\right)}^{1/q}\le {\left(\sum _{i=0}^n(x_{i+1}-x_i{)}^{q-q/p+1}\Vert f_i^{\prime }{\Vert }_p^q\right)}^{1/q}.$$

$$\Vert f{\Vert }_q$$

$$={\ell }^{1-1/p+1/q}\Vert f^{\prime }{\Vert }_p={\ell }^{1-1/p+1/q}.$$

$$\Vert f{\Vert }_q$$

$$={\left(\sum _{i=0}^n{\ell }_i^{s+1}\right)}^{1/s}.$$

$$\mathbb{E}\bigg{[}\Big{(}\sum_{i=0}^{n}\ell_{i}^{s+1}\Big{)}^{1/s}\bigg{]}\asymp\frac{1}{n}$$

$$\mathbb{E}\bigg{[}\Big{(}\sum_{i=0}^{n}\ell_{i}^{s+1}\Big{)}^{1/s}\bigg{]}\leq\bigg{(}\mathbb{E}\Big{[}\sum_{i=0}^{n}\ell_{i}^{s+1}\Big{]}\bigg{)}^{1/s}\qquad\text{whenever }s\geq 1.$$

$$\mathbb{E}\Big{[}\sum_{i=0}^{n}\ell_{i}^{s+1}\Big{]}\asymp n^{-s}$$

$$\mathbb{E}\Big{[}\sum_{i=0}^{n}h(\ell_{i})\Big{]}=n(n+1)\int_{0}^{1}(1-r)^{n-1}h(r){\rm d}r$$

$$\mathbb{E}\Big{[}\sum_{i=0}^{n}\ell_{i}^{s+1}\Big{]}=n(n+1)B(s+2,n)=\frac{(n+1)!(s+1)!}{(n+s+1)!}\asymp n^{-s},$$

$$\mathbb{E}[r(N_n)]\asymp \{\begin{array}{ll}r(N_n^{\ast })\asymp n^{-1}& \text{if}p>1\\ & \\ r(N_{n/\log n}^{\ast })\asymp {\left(\frac{n}{\log n}\right)}^{-1}& \text{if}p=1.\end{array}$$

$$\frac{c\log n}{n}\le \mathbb{E}[r(N_n)]\le \frac{8\log n}{n}$$

$$F_{d}=\Big{\{}f:[0,1]^{d}\to\mathbb{R}\,:\,\left|f(x)-f(y)\right|\leq\operatorname{dist}(x,y)\,\,\,\forall x,y\in[0,1]^{d}\Big{\}}.$$

$$\mathrm{dist}(x,y)=\underset{k\in {\mathbb{Z}}^d}{\min }{\Vert x+k-y\Vert }_{\infty }\text{for}x,y\in [0,1{]}^d.$$

$$N_n(f)=(f(x_1),\dots ,f(x_n)),x_1,\dots ,x_n\in [0,1{]}^d.$$

$$f^{+}(x)=\min_{i\leq n}\big{(}f(x_{i})+\operatorname{dist}(x,x_{i})\big{)}\quad\text{and}\quad f^{-}(x)=\max_{i\leq n}\big{(}f(x_{i})-\operatorname{dist}(x,x_{i})\big{)}.$$

$$r(N_n)=\underset{f\in F_d}{\sup }{\Vert f-A_n(f)\Vert }_q=\underset{\begin{array}{c}f\in F_d\\ N_n(f)=0\end{array}}{\sup }{\Vert f\Vert }_q.$$

Full text

1. Introduction

Optimal algorithms use optimal information about the problem instances. Quite often, we do not have access to optimal information. One reason may be that we do not know how to choose the most significant measurements. In this paper, we assume that information comes in randomly, which is a standard assumption in learning theory and uncertainty quantification. We ask the following question:

What is the typical quality of random information?

Of course, random information cannot be better than optimal information. In many cases, however, it will turn out that random information is practically as good as optimal information while in other situations random information is a little worse. Sometimes, random information is completely useless.

To phrase our question precisely, we need to clarify how we measure the quality of information and what we mean by random. The first is done with the so-called radius of information, which is the worst case error of the best algorithm that uses nothing but the given information and the a priori knowledge about the problem instance. Random information, on the other hand, shall be obtained from a certain number of independent measurements that all follow the same law. We study the question for two different natural choices of uniform distributions.

Let us go a little more into detail. A linear problem is given by a linear solution operator $S$ that maps from a convex and symmetric subset $F$ of a normed space to a normed space $G$ and a class $\Lambda$ of continuous linear functionals on $F$, the class of admissible measurements. We may think of an integration problem, where $S(f)$ is the integral of a function $f$, or a recovery problem, where $S$ is an embedding. One wants to approximate the solution $S(f)$ for unknown $f\in F$ based on $n$ of these measurements such that we can guarantee a small error with respect to the norm in $G$. We refer the reader to [29, 36] for a detailed exposition. An information mapping has the form

[TABLE]

where $L_{1},\dots,L_{n}\in\Lambda$. The power or quality of the information mapping is measured by the radius of information. This is the worst case error of the best algorithm $A_{n}={\varphi}\circ N_{n}$ based on this information, i.e.,

[TABLE]

We say that information is optimal, and write $N_{n}^{*}$ instead of $N_{n}$, if

[TABLE]

Here we study random information of the form (1), where the random functionals $L_{1},\dots,L_{n}\in\Lambda$ are independent and identically distributed. The goal is to compare

[TABLE]

the radius of optimal information and the expected radius of random information.

If the infimum and the expected value are comparable, this means that there are many good algorithms based on many different information mappings. In this case, optimal information and therefore optimal algorithms are not very special. On the other hand, if the infimum is significantly smaller than the expected value, this means that optimal information is indeed special. It seems to be an interesting characteristic of a problem whether optimal information is special or not. Of course, the results of our comparisons heavily depend on the distribution underlying the measurements. While the question may be interesting for many distributions, we feel that there often is a natural choice. If the distribution only depends on the class $\Lambda$ of admissible measurements, collecting random information might even be a good idea if optimal information is available. It may happen that we do not loose much in terms of the radius but gain the following useful properties:

• •

Since the distribution is independent of $n$, it is easy to increase the number of measurements if our current approximation is not yet satisfactory.

• •

The information can be used for many different input classes $F$, solution operators $S$, and target spaces $G$. In that sense it is universal.

The second property does not mean that the corresponding algorithm $A_{n}={\varphi}\circ N_{n}$ is universal. The optimal choice of ${\varphi}$ may depend on $F$, $S$, and $G$. It is well known, see again [36], that any interpolatory algorithm is optimal up to a factor two.

We note that examples of the sort ‘random information is good’ can be deduced from work using the probabilistic method to prove upper bounds of $r(N_{n}^{*})$. Here one uses upper bounds of $\mathbb{E}[r(N_{n})]$ and the trivial inequality $r(N_{n}^{*})\leq\mathbb{E}[r(N_{n})]$, see [17, 29, 31, 32, 33] for many such results. The idea is to introduce a random family of algorithms and to show that the expected worst case error is small. This is used to obtain the existence of good algorithms. However, it actually implies that most of the algorithms in that family are good. Therefore, the expected radius of the random information underlying these algorithms must be small as well. We note that for this approach only upper bounds of $\mathbb{E}[r(N_{n})]$ are needed and lower bounds are usually not studied.

2. Uniformly distributed function evaluations

If the input class $F$ consists of functions, one natural restriction of the class of information functionals is so called standard information consisting of function evaluations. Moreover, if the domain of definition of functions in $F$ carries a probability measure, a natural source of random information are function values $f(x)$ with $x$ distributed according to the probability measure. In this section, we analyze this setting for different input classes $F$ of smooth functions defined on the unit cube $[0,1]^{d}$. Then the random information consists of function values $f(x)$ with $x$ uniformly distributed in $[0,1]^{d}$.

2.1. Integration and Approximation in univariate Sobolev spaces of smoothness 1

We consider the $L_{q}$-approximation problem

[TABLE]

given by $\operatorname{APP}(f)=f$ using function values as information. The Sobolev spaces $W_{p}^{1}([0,1])$ are usually normed by

[TABLE]

It is well known that the optimal information is given by

[TABLE]

i.e., point evaluations at equidistant points. The minimal radius of information then satisfies

[TABLE]

see (6).

Here and later we use asymptotic notation as follows. For sequences $(a_{n})$ and $(b_{n})$ of positive real numbers, we write $a_{n}\preccurlyeq b_{n}$ and $b_{n}\succcurlyeq a_{n}$ to indicate that there exists a constant $C\in(0,\infty)$ such that $a_{n}\leq C\,b_{n}$ for all $n$. We shall write $a_{n}\asymp b_{n}$ if $a_{n}\preccurlyeq b_{n}$ and $b_{n}\preccurlyeq a_{n}$. Sometimes we use similar notation for double sequences.

We will show that information given by $n$ independent and uniformly distributed points in $[0,1]$ is as good as optimal information provided that $p>q$, whereas in the case $p\leq q$ there is a loss of a logarithmic factor.

Theorem 1**.**

Consider $L_{q}$-approximation of functions from $W^{s}_{p}([0,1])$ with $1\leq p,q\leq\infty$, using function values at uniformly distributed points in $[0,1]$.

[TABLE]

The proof of Theorem 1 is based on the following formula for the radius of information $r(N_{n})$ for information N_{n}(f)=\big{(}f(x_{1}),f(x_{2}),\dots,f(x_{n})\big{)}, where we assume $0=x_{0}\leq x_{1}\leq\dots\leq x_{n}\leq x_{n+1}=1$.

Lemma 2**.**

[TABLE]

Proof.

Note that in this case the radius of information satisfies

[TABLE]

with an equivalence constant of at most 2, see [29, Section 4.2]. We start with the lower bound in the case $p\leq q$. Let $i_{0}\in\{0,1,\dots,n\}$ be such that

[TABLE]

and consider a continuous piecewise linear function $f:[0,1]\to\mathbb{R}$ that is zero outside of the interval $(x_{i_{0}},x_{i_{0}+1})$, satisfies $f^{\prime}(x)=1$ in the left half of this interval and $f^{\prime}(x)=-1$ in the right half. Then

[TABLE]

show that $r(N_{n})\succcurlyeq\ell^{1-1/p+1/q}$.

We turn to the lower bound in the case $p>q$. In this case, let $f:[0,1]\to\mathbb{R}$ be a continuous piecewise linear function satisfying $f(x_{i})=0$ for $i=0,1,\dots,n+1$ and $f^{\prime}(x)=\ell_{i}^{\alpha}$ in the left half of each interval $[x_{i},x_{i+1}]$ and $f^{\prime}(x)=-\ell_{i}^{\alpha}$ in the right half, where $\ell_{i}=x_{i+1}-x_{i}$ and $\alpha=q/(p-q)$. Then $N_{n}(f)=(0,\dots,0)$ and

[TABLE]

show that $r(N_{n})\succcurlyeq\left(\sum_{i=0}^{n}\ell_{i}^{(pq+p-q)/(p-q)}\right)^{1/q-1/p}.$

Finally, we prove the upper bound. Observe that $\|f\|_{\infty}\leq\|f^{\prime}\|_{1}$ holds for any function $f\in W_{1}^{1}([0,1])$ with $f(0)=0$. Consequently, $\|f\|_{q}\leq\|f^{\prime}\|_{p}$ holds for any function $f\in W_{p}^{1}([0,1])$ with $f(0)=0$. By scaling, it follows that $\|f\|_{q}\leq(b-a)^{1-1/p+1/q}\|f^{\prime}\|_{p}$ holds for any function $f\in W_{p}^{1}([a,b])$ satisfying either $f(a)=0$ or $f(b)=0$. Let now $f\in W_{p}^{1}([0,1])$ be such that N_{n}(f)=\big{(}f(x_{1}),f(x_{2}),\dots,f(x_{n})\big{)}=(0,\dots,0) and $\|f^{\prime}\|_{p}=1$. Let $f_{i}$ be the restriction of $f$ to the interval $[x_{i},x_{i+1}]$. Then

[TABLE]

In the case $p\leq q$, let again $\ell$ be the length of the largest of the intervals $[x_{i},x_{i+1}]$. It follows that

[TABLE]

This implies $r(N_{n})\leq\ell^{1-1/p+1/q}$ as claimed. In the case $p>q$, let again $\ell_{i}=x_{i+1}-x_{i}$ and $s>0$ be given by $\frac{1}{s}=\frac{1}{q}-\frac{1}{p}$. Using Hölder’s inequality, we arrive at

[TABLE]

This implies the claimed upper bound also when $p>q$. ∎

Proof of Theorem 1.

In the case $p\leq q$ the proof follows from Lemma 2 and the fact that, in expectation, the largest gap among $n$ independent and uniformly distributed points in $[0,1]$ is of order $\log(n)/n$. This is a simple consequence of Lemma 7. In the case $p>q$ the proof follows from

[TABLE]

for any $s>0$, where the $\ell_{i}$ are again the spacings of independent and uniformly distributed points in $[0,1]$. In fact, the lower bound follows trivially since the expected radius of random information is larger than the radius of optimal information. To prove the upper bound observe first that the function $s\mapsto\left(\sum_{i=0}^{n}\ell_{i}^{s+1}\right)^{1/s}$ is non-decreasing, since $\sum_{i=0}^{n}\ell_{i}=1$, and that Jensen’s inequality implies

[TABLE]

Hence, it is enough to show that

[TABLE]

for positive integers $s$. It was proved in [9] that

[TABLE]

for any integrable function $h:[0,1]\to\mathbb{R}$. With $h(r)=r^{s+1}$, we obtain

[TABLE]

where $B(x,y)$ is the Beta function. This completes the proof. ∎

Remark 3**.**

For the classes studied, i.e., when the smoothness is one, the radius of information $r(N_{n})$ for the integration problem with $S(f)=\int_{0}^{1}f(x)\,{\rm d}x$ is the same as for the $L_{1}$-approximation problem. Hence, for the integration problem for functions in $W_{p}^{1}([0,1])$, we have

[TABLE]

Some cases of Theorem 1 are known, sometimes in a slightly different language since authors had in mind discrepancy (with optimal weights) instead of the integration problem. The star discrepany for $d=1$, i.e., $p=q=1$, was studied in [38]. Here we have $r(N_{n}^{*})=\frac{1}{2n}$. From Corollary 3.3 and Theorem 3.5 of [38] we know that

[TABLE]

for some $c\in(0,\infty)$. Therefore, random information is good, but not optimal. Also the case $p=2$ and $q=1$ is covered by [38, Theorems 4.2 and 4.3].

2.2. Approximation of multivariate Lipschitz functions

We study the problem of $L_{q}$-approximation of functions from the class

[TABLE]

Here, $\operatorname{dist}$ denotes the maximum metric on the $d$-torus, i.e.,

[TABLE]

We consider standard information of the form

[TABLE]

An optimal algorithm based on $N_{n}$ works as follows:

Algorithm. For $N_{n}$ as above and $f\in F_{d}$, we define $A_{n}(f)=(f^{+}+f^{-})/2$, where

[TABLE]

Note that $f^{+}$ and $f^{-}$ are the maximal and the minimal function in $F_{d}$ that interpolate $f$ at the nodes $x_{1},\dots,x_{n}$. We have the following relations that we prove for the sake of completeness, see e.g. [29, 36].

Lemma 4**.**

[TABLE]

Proof.

Clearly $A_{n}$ is of the form ${\varphi}\circ N_{n}$ for some mapping ${\varphi}:\mathbb{R}^{n}\to L_{q}$. The definition of the radius yields

[TABLE]

On the other hand, any $f\in F_{d}$ satisfies the pointwise estimate

[TABLE]

This implies

[TABLE]

since the right-hand side of (5) is an element of $F_{d}$ that vanishes on $\{x_{1},\ldots,x_{n}\}$. Altogether, we obtain the claimed identities. ∎

It is also well known that optimal information is given by function values on a regular grid if $n=m^{d}$ for some $m\in\mathbb{N}$. The radius of optimal information $N_{n}^{*}$ satisfies

[TABLE]

for all $1\leq q\leq\infty$. This follows from the upper bound on the complexity of uniform approximation as studied in [34] and the lower bound on the complexity of numerical integration as studied in [35]. Using the technique of proof of the latter, we even obtain a precise formula for the radius of optimal information.

Proposition 5**.**

Let $n=m^{d}$ for some $m\in\mathbb{N}$. Then

[TABLE]

Proof.

Let $N_{n}$ be as above and let $P_{n}=\{x_{1},\ldots,x_{n}\}$. Note that the function $\operatorname{dist}(\cdot,P_{n})$ is contained in $F_{d}$ and vanishes on $P_{n}$. On the other hand, every other function $f\in F_{d}$ that vanishes on $P_{n}$ must satisfy

[TABLE]

This yields

[TABLE]

Let us first consider the case $q=\infty$. Since the volume of the union of the balls $B_{r}^{\infty}(x)$ over $x\in P_{n}$ is smaller than 1 for all $r<1/(2m)$, there must be some $x\in[0,1]^{d}$ with $\operatorname{dist}(x,P_{n})\geq r$. Therefore,

[TABLE]

where equality holds when $P_{n}=\{i/m\,:\,0\leq i<m\}^{d}$. Let us turn to the case $q<\infty$. Let $\lambda^{d}$ denote Lebesgue measure on $[0,1]^{d}$. Then

[TABLE]

Note that

[TABLE]

where equality holds if the sets $B_{t^{1/q}}^{\infty}(y)$ for $y\in P_{n}$ are pairwise disjoint. Hence,

[TABLE]

with equality for $P_{n}=\{i/m\,:\,0\leq i<m\}^{d}$. This proves the statement. ∎

In the following, we study the expected radius of random information

[TABLE]

where the points $x_{1},\ldots,x_{n}$ are independent and uniformly distributed in $[0,1]^{d}$. If $q$ is finite, the $q^{\rm th}$ moment of the radius at zero can be computed precisely.

Proposition 6**.**

Consider $L_{q}$-approximation of functions from $F_{d}$ defined in (4) with $1\leq q<\infty$, using function values at uniformly distributed points in $[0,1]^{d}$. Then

[TABLE]

In particular, we have

[TABLE]

Proof.

Let $P_{n}=\{x_{1},\ldots,x_{n}\}$. Recall that

[TABLE]

Using Tonelli’s theorem, we obtain

[TABLE]

We will show that the integrand of the latter integral is constant. Let us fix $x\in[0,1]^{d}$ and note that $\operatorname{dist}(x,P_{n})\in[0,1/2]$. For any $t\in[0,1/2]$, we have

[TABLE]

Thus

[TABLE]

The substitution $s=1-(2t^{1/q})^{d}$ and integration by parts yield

[TABLE]

which implies the statement of the theorem. ∎

We now turn to the case of uniform approximation, i.e., $q=\infty$. In this case, the expected radius of information is closely related to the so called coupon collector’s problem. This is the question for the random number $\tau_{\ell}$ of coupons that a coupon collector has to collect to obtain a complete set of $\ell$ distinct coupons. The following facts on the distribution of $\tau_{\ell}$ are well known, see [27]. Here $H_{\ell}=\sum_{k=1}^{\ell}1/k$ is the $\ell^{th}$ harmonic number. Note that $H_{\ell}\sim\log\ell$ as $\ell\to\infty$.

Lemma 7**.**

Let $(Y_{i})_{i=1}^{\infty}$ be a sequence of random variables that are uniformly distributed in the set $\left\{1,\ldots,\ell\right\}$ and let

[TABLE]

Then

[TABLE]

and for any $c\in(0,\infty)$,

[TABLE]

This leads to the following estimates of the expected radius for $q=\infty$.

Proposition 8**.**

Consider $L_{\infty}$-approximation of functions from $F_{d}$ defined in (4). For $n\in\mathbb{N}$, let

[TABLE]

Then

[TABLE]

Proof.

Let $m\in\mathbb{N}$ and decompose $[0,1]^{d}$ into $\ell=m^{d}$ boxes

[TABLE]

of equal volume. Again, let $P_{n}=\{x_{1},\ldots,x_{n}\}$. Recall that the radius of the information N_{n}(f)=\big{(}f(x_{1}),\dots,f(x_{n})\big{)} is given by

[TABLE]

Therefore, $r(N_{n})$ is bounded above by $1/m$ if every box contains a point of $P_{n}$, and bounded below by $1/(2m)$ if one of the boxes does not contain a point of $P_{n}$. Let $A$ be the event that every box contains a point. Note that the number of random points $x_{i}$ that it takes to hit all the boxes follows the distribution of the coupon collector’s variable $\tau_{\ell}$ as defined in Lemma 7. For the upper bound, we choose $m=m_{2}$. Lemma 7 yields

[TABLE]

and hence

[TABLE]

For the lower bound, we choose $m=m_{1}$. Chebyshev’s inequality yields

[TABLE]

We obtain

[TABLE]

as it was to be proven. ∎

We arrive at the following result on the power of random information proved in [25]. Note that the case $q=\infty$ is already known from [2], where the authors study the uniform approximation of functions on $[0,1]^{d}$ with bounded $r^{\rm th}$ derivative.

Theorem 9** (Krieg).**

Consider $L_{q}$-approximation of functions from $F_{d}$ defined in (4) with $1\leq q\leq\infty$, using function values at uniformly distributed points in $[0,1]^{d}$. Then

[TABLE]

Proof.

The statements on the rate of the optimal radius of information follow from Proposition 5. The upper bound on the expected radius in the first case follows from Proposition 6, since the expected radius is bounded above by its $q^{\rm th}$ moment. The lower bound is trivial. The upper and lower bound on the expected radius in the second case follow from Proposition 8, since both $m_{1}^{d}$ and $m_{2}^{d}$ are of order $n/\log n$. ∎

Thus, in the sense of order of convergence, random information is as good as optimal information for the problem of $L_{q}$-approximation on $F_{d}$ if $q<\infty$. If $q=\infty$, we loose a logarithmic factor.

Remark 10** (Modifications of $F_{d}$).**

The rates of convergence of the expected and the optimal radius do not change if we replace the maximum metric on the torus by some equivalent metric. The same holds true if we change the Lipschitz constant or if we switch to the non-periodic setting.

2.3. Classical Sobolev spaces

We study the general $L_{q}$-approximation problem for Sobolev spaces,

[TABLE]

given by $\operatorname{APP}(f)=f$ with standard information. A typical choice for the norm in this space is

[TABLE]

It is known that the radius of optimal information $N_{n}^{*}$ satisfies

[TABLE]

where we assume that $s>d/p$ holds, to ensure that the functions are continuous. This can be found in [28] for general Lipschitz domains and was known earlier for special cases, such as the cube, see [28, Remark 24]. Here we ask for the expected radius of random information $N_{n}$ given by $n$ independent and uniformly distributed points in $[0,1]^{d}$. We conjecture that random information is as good as optimal information provided that $p>q$, whereas in the case $p\leq q$ there is a loss of a logarithmic factor. This conjecture is true in the following special cases:

• •

Section 2.1 covers univariate Sobolev spaces of smoothness $1$, i.e., $s=d=1$.

• •

Section 2.2 covers Lipschitz functions, i.e., $s=1$ and $p=\infty$.

• •

The paper [2] covers uniform approximation on Hölder classes, i.e., $p=q=\infty$.

In the general case, we are only able to prove the second part of our conjecture.

Theorem 11**.**

Consider $L_{q}$-approximation of functions from $W^{s}_{p}([0,1]^{d})$ with $1\leq p\leq q\leq\infty$, using function values at uniformly distributed points in $[0,1]^{d}$. Then

[TABLE]

We use the following criterion for optimal point sets, which can be found in [28].

Lemma 12**.**

Consider $L_{q}$-approximation of functions from $W^{s}_{p}([0,1]^{d})$ with $1\leq p,q\leq\infty$. For all $n\in\mathbb{N}$, let $P_{n}\subset[0,1]^{d}$ be a point set of cardinality $n$,

[TABLE]

If $r_{n}\preccurlyeq d_{n}$, then the information $N_{n}$ with nodes $P_{n}$ satisfies $r(N_{n})\asymp r(N_{n}^{*})$.

Proof of Theorem 11.

Let $\alpha=s/d-1/p+1/q$. We start with the lower bound. Let $g:\mathbb{R}^{d}\to\mathbb{R}$ be a smooth function with support in the $\ell_{\infty}$-unit ball that is not identically zero. Using Lemma 7, we obtain with probability $1/2$ that $[0,1]^{d}$ contains an $\ell_{\infty}$-ball of radius

[TABLE]

not containing any of the points in $P_{n}$. If $x_{0}$ denotes the center of this ball, then the function $f\in W_{p}^{s}([0,1]^{d})$ given by $f(x)=g(\tau_{n}^{-1}(x-x_{0}))$ has support in this ball and satisfies

[TABLE]

This yields

[TABLE]

We turn to the upper bound. We choose $m\in\mathbb{N}_{0}$ maximal such that $\ell=(3m)^{d}$ satisfies

[TABLE]

We split the unit cube into $k=m^{d}$ subcubes of equal volume. Moreover, we split each of these into $3^{d}$ subcubes of equal volume. This gives us $\ell=(3m)^{d}$ small cubes. Let $A$ be the event that we hit every small cube. Lemma 7 yields

[TABLE]

Assume now that $A$ takes place. We choose exactly one point out of the small cube in the center of every large cube to obtain a subset $P_{k}$ of $P_{n}$ with cardinality $k$. Then

[TABLE]

Let $N_{k}^{\prime}$ be the information map with nodes $P_{k}$. Lemma 12 yields

[TABLE]

Together, (7) and (8) imply the statement. ∎

We note that the authors of [12] study the integration problem for very general Sobolev spaces on closed manifolds. One can obtain the optimal rate $r(N_{n}^{*})\asymp n^{-s/d}$. The authors prove the upper bound

[TABLE]

where the random points are chosen with respect to the probability measure that defines the integral to be approximated. The given proof works with the covering radius of random points which is an important characteristic for $q=\infty$. For the integration problem or $q=1$, we expect, however, that the additional log factor is needed only for $p=1$.

Remark 13**.**

Another important case are tensor products of univariate Sobolev spaces. These spaces are usually called Sobolev spaces with (dominating) mixed smoothness or in the periodic case sometimes Korobov spaces. One of the main features is that the optimal (main) orders of convergence are independent of the dimension, only the logarithmic terms depend on the dimension. We refer to [11] for a survey. There exist upper bounds for the numbers $\mathbb{E}[r(N_{n})]$, but these bounds are weaker than for isotropic Sobolev spaces. One reason is that Lemma 12 does not hold for these spaces and the geometrical structure of good sampling points is not known. See [1, 17, 20, 22, 29, 31, 32] for upper bounds on $\mathbb{E}[r(N_{n})]$ in various cases.

Remark 14**.**

Recovery of functions of many variables usually suffers from the curse of dimensionality, even for smooth $C^{\infty}$ functions, see [30]. To avoid the curse one can impose structural properties, such as the sparsity with respect to the Fourier coefficients or another orthonormal system. Then one can find an approximation of the function with relatively few random function values, see [6, 8]. Another way to study structural properties is to define weighted norms as was first done in [33], see [29, 31, 32] for a survey.

3. Uniformly distributed linear information

In this section we study linear information consisting of arbitrary continuous linear functionals. We are going to present two recovery problems for which we compare random information with optimal information. The first is a classical result due to Kashin, Garnaev, and Gluskin [24, 14] on the recovery of $\ell_{1}^{m}$-vectors in the Euclidean distance, while the second are novel results of the authors on the recovery of vectors in an $m$-dimensional ellipsoid in the Euclidean distance [18].

3.1. Recovery of $\ell_{1}^{m}$-vectors in the Euclidean norm

We study the problem of recovery of vectors in the unit ball of $\ell_{1}^{m}$ with general linear information and error measured in the Euclidean norm. This is the approximation problem with $F=\ell_{1}^{m}$ and $G=\ell_{2}^{m}$, i.e., $S$ is the identity map $\ell_{1}^{m}\hookrightarrow\ell_{2}^{m}$. The minimal worst case error, which is the minimal radius of information achievable with general linear information, is closely connected to the notion of Gelfand numbers or Gelfand widths. Gelfand numbers and their dual counterpart Kolmogorov numbers are fundamental concepts in classical and modern approximation and complexity theory.

For $n\in\mathbb{N}$, the $n^{\rm th}$ Gelfand number of a linear bounded operator $S:F\to G$ between normed spaces $F$ and $G$ can be defined as

[TABLE]

If the subspace $F_{n}$ of $F$ is the kernel of a linear information map $N_{n}:S\to\mathbb{R}^{n}$, then

[TABLE]

is the local radius of zero information. It is not hard to see that

[TABLE]

Since every subspace $F_{n}$ of codimension at most $n$ is the kernel of a linear information map $N_{n}:S\to\mathbb{R}^{n}$, we have the well-known relation

[TABLE]

between the radius of the optimal information $N_{n}^{\ast}$ for the linear problem $S:F\to G$ and the Gelfand numbers $c_{n}(S)$.

The systematic study of the Gelfand numbers of identity maps $\ell_{p}^{m}\hookrightarrow\ell_{q}^{m}$ has a long tradition and is the basis for the study of Gelfand numbers of embeddings of Sobolev spaces and, therefore, also for the complexity of $L_{q}$-approximation of functions from Sobolev spaces. We want to discuss here the case $p=1$ and $q=2$ which was one of the difficult cases. Its solution is an example for the power of random information. While lower bounds had been obtained before, the breakthrough regarding the upper bounds was made using random approximation, a groundbreaking method having its origin in the work of Kashin [24]. The problem was finally settled in the series of papers by Kashin [23, 24], Gluskin [15, 16], and Garnaev and Gluskin [14].

Let us consider now the particular case of the identity $\ell_{1}^{m}\hookrightarrow\ell_{2}^{m}$, which is of importance in information based complexity since it also demonstrates that non-linear recovery algorithms can be much better than linear ones. Although the results in the literature are usually formulated in the language of Gelfand numbers, we discuss the results directly for the radius of information.

Recall that we now allow arbitrary linear information $N_{n}:\ell_{1}^{m}\to\mathbb{R}^{n}$. The random information we consider is Gaussian information, where $N_{n}=N_{n,m}$ is given by an $n\times m$-matrix with independent standard normal entries. The kernels of these maps are distributed according to the Haar measure on the Grassmannian manifold of $n$-codimensional subspaces in $\mathbb{R}^{m}$. In this sense, we can speak about uniformly distributed linear information.

In [24, Theorem 1], Kashin introduced his fundamentally new approach via random subspaces and obtained that

[TABLE]

Through a refinement of Kashin’s arguments, Garnaev and Gluskin [14] were able to decrease the power $3/2$ to a square root, while at the same time providing the matching lower bounds. Altogether, we have the following result.

Theorem 15** (Kashin, Garnaev, Gluskin).**

Consider the recovery problem of vectors from $\ell_{1}^{m}$ in the Euclidean norm and Gaussian information. Then

[TABLE]

We want to sketch the proof of the upper bound

[TABLE]

of Theorem 15 and demonstrate the power of the probabilistic method of Kashin, Garnaev, and Gluskin.

Step 1. As already explained, one considers a Gaussian random matrix $G=(g_{ij})_{i,j=1}^{n,m}$ with independent and identically distributed standard normal entries and defines the random information mapping

[TABLE]

Step 2. One defines the mapping ${\varphi}$ to be

[TABLE]

and considers the corresponding random algorithm $A_{n}={\varphi}\circ N_{n}$.

Step 3. One can now prove that with high probability the error of the random algorithm is small for any $x\in\mathbb{B}_{1}^{m}$. More precisely, if $\delta\in(0,\infty)$, then taking

[TABLE]

yields a worst case error

[TABLE]

for the random algorithm $A_{n}$ with probability at least $1-\delta$.

As a matter of fact, one actually shows that most linear information mappings $N_{n}:\mathbb{R}^{m}\to\mathbb{R}^{n}$ lead to a small error and so random information is essentially as good as optimal information. We want to emphasize that no explicit construction for such a mapping is known. So the optimal information is not accessible in this case and one has to settle for random information. We also want to mention that linear algorithms corresponding to linear widths or approximation numbers are much worse than nonlinear algorithms for this recovery problem.

Let us mention that Garnaev and Gluskin actually obtained sharp bounds in the more general setting of recovery of vectors from $\ell_{1}^{m}$ in the $\ell_{q}^{m}$-norm or Gelfand numbers of the identity $\ell_{1}^{m}\hookrightarrow\ell_{q}^{m}$. If the random information $N_{n,m}$ is again Gaussian information and $1<q\leq 2$, then

[TABLE]

Later, Gelfand widths have also attracted quite some attention in the area of compressive sensing over the last two decades, see, e.g., [3, 4, 7, 10]. The goal of compressive sensing is to recover compressible vectors $x\in\mathbb{R}^{m}$ (those which are close to sparse vectors with only few non-zero coordinates) from $n$ pieces of incomplete linear information. In fact, vectors in the unit ball of $\ell_{p}^{m}$ with $0<p\leq 1$ serve as a good model for sparse vectors. In this context, also the recovery of vectors in $\ell_{p}^{m}$ with $0<p\leq 1$ or the Gelfand numbers of the identity $\ell_{p}^{m}\hookrightarrow\ell_{q}^{m}$ were studied. Donoho [10] and Foucart, Pajor, Rauhut, and T. Ullrich [13, Theorem 1.1] extended the Garnaev-Gluskin result also to the case $0<p\leq 1$ and $p<q\leq 2$ and proved

[TABLE]

Again, it is not known how two explicitly construct measurement maps performing almost as good as $N_{n,m}^{\ast}$.

A further problem in compressed sensing is the matrix recovery problem. The analogon to the recovery problem of vectors from $\ell_{p}^{m}$ in the $\ell_{q}^{m}$-norm is the recovery problem of matrices from Schatten $p$-classes in the Schatten $q$-norm. The particular case $q=2$ is is the Frobenius norm. This problem was studied in [5, 21].

3.2. Recovery of vectors from an ellipsoid

We continue with another recovery problem and recent results of the authors concerning the quality of random information for $\ell_{2}$-approximation, see [18].

Let $F$ be a centered ellipsoid in $\mathbb{R}^{m}$ with semi-axes $\sigma_{1}\geq\dots\geq\sigma_{m}$. Clearly, $F$ can be seen as the unit ball of some Hilbert space. We want to recover $x\in F$ from the data $N_{n}(x)\in\mathbb{R}^{n}$ with linear information mapping $N_{n}\in\mathbb{R}^{n\times m}$ and measure the error in the Euclidean distance. While the power of the information mapping is

[TABLE]

it is known that, for linear problems in Hilbert spaces, the worst data is the zero data and so

[TABLE]

where $E_{n}$ is the kernel of $N_{n}$, see [29, 36]. In this case, the power or radius of optimal information is given by $r(N_{n}^{*})=\sigma_{n+1}$. The question is now how good random information $N_{n}$ is for the $\ell_{2}$-recovery problem, when random information is provided by a Gaussian random matrix $G=(g_{ij})_{i,j=1}^{n,m}$? Is it comparable to optimal information?

As is explained in [18], rephrased in geometric terms, one is interested in the circumradius of the intersection of a centered ellipsoid $F$ in $\mathbb{R}^{m}$ with a random subspace $E_{n}$ of codimension $n$. While the maximal radius is the length of the largest semi-axis $\sigma_{1}$, the minimal radius is the length of the $(n+1)^{\rm st}$ largest semi-axis $\sigma_{n+1}$. But how large is the radius of a typical intersection? Is it comparable to the minimal or the maximal radius or does it behave completely different?

Using various probabilistic tools, such as exponential estimates for sums of chi-squared random variables, Gordon’s min-max theorem for Gaussian processes, or estimates for the extreme singular values of (structured) Gaussian matrices, the authors obtained upper and lower bounds on the radius of Gaussian information for the $\ell_{2}$-approximation problem which hold with overwhelming probability. In many cases these bounds are sharp up to absolute constants and show that random information is comparable to optimal information.

Let us only present one case which is of particular importance, where the semi-axes behave like the singular values of Sobolev embeddings [18, Corollary].

Theorem 16** (Hinrichs, Krieg, Novak, Prochno, Ullrich).**

For $k\in\mathbb{N}$ let

[TABLE]

where $\alpha\in(0,\infty)$ and $\beta\in\mathbb{R}$. Then

[TABLE]

with

[TABLE]

where $c\in(0,\infty)$ is an absolute constant.

This means that random information is just as good as optimal information if the singular values decay with a polynomial rate greater than $1/2$. Then, in geometric terms, the size of a typical intersection ellipsoid is comparable to the size of the smallest intersection. On the other hand, if the singular values decay too slowly, random information is rather useless, which means that a typical intersection ellipsoid is almost as large as the largest.

The ellipsoid recovery problem can be considered in an infinite dimensional setting as well, even though our geometric intuition might fail. In this case, where $m=\infty$, we consider the ellipsoid

[TABLE]

and the matrix $(g_{ij})_{1\leq i\leq n,j\in\mathbb{N}}$ with independent standard Gaussian entries as random information mapping $N_{n}$.

As a consequence of the probabilistic estimates derived in [18, Theorems 3 and 4], one obtains an $\ell_{2}$-dichotomy showing that random information is useful if and only if $\sigma\in\ell_{2}$ [18, Corollary 5].

Theorem 17** (Hinrichs, Krieg, Novak, Prochno, Ullrich).**

If $\sigma\not\in\ell_{2}$, then $r(N_{n})=\sigma_{1}$ holds almost surely for all $n\in\mathbb{N}$. If $\sigma\in\ell_{2}$, then

[TABLE]

Remark 18**.**

Such an $\ell_{2}$-dichotomy is known from a related problem. There $F$ is the unit ball of a reproducing kernel Hilbert space $H$, i.e., $H\subset L_{2}(D)$ consists of functions on a common domain $D$ and the evaluation functional $f\mapsto f(x)$ is a bounded operator on $H$ for every $x\in D$. The optimal linear information $N_{n}$ for the $L_{2}$-approximation problem is given by the singular value decomposition and has radius $\sigma_{n+1}$. This information might be difficult to implement and hence one might allow only information $N_{n}$ of the form

[TABLE]

The goal is to relate the power of function evaluations to the power of all continuous linear functionals and one would like to prove that their power is roughly the same. Unfortunately, this is not true in general. When $\sigma\notin\ell_{2}$ the convergence of optimal algorithms that may only use function values can be arbitrarily slow [19]. The situation is much better if we assume that $\sigma\in\ell_{2}$. It was shown in [26] and [37] that function values are almost as good as general linear information. We refer to [32, Chapter 26] for a presentation of these results.

Acknowledgement

The authors would like to thank the Isaac Newton Institute for Mathematical Sciences for support and hospitality during the programme ‘Approximation, sampling and compression in data science’ when work on this paper was undertaken. This work was supported by EPSRC Grant Number EP/R014604/1.

A. Hinrichs and J. Prochno are supported by the Austrian Science Fund (FWF) Projects F5509-N26 and F5508-N26, which are part of the Special Research Program “Quasi-Monte Carlo Methods: Theory and Applications”. A. Hinrichs was also supported by a grant from the Simons Foundation. J. Prochno is also supported by a Visiting Professor Fellowship of the Ruhr University Bochum and its Research School PLUS.