Can we trust Bayesian uncertainty quantification from Gaussian process priors with squared exponential covariance kernel?

TL;DR

This paper critically examines the reliability of Bayesian credible sets derived from Gaussian process priors with squared exponential kernels, revealing overconfidence issues and proposing modifications for improved uncertainty quantification.

Contribution

It demonstrates that standard hyper-parameter estimation leads to overconfident credible sets and proposes logarithmic adjustments to achieve reliable, adaptive uncertainty quantification.

Findings

Credible sets are overconfident with maximum marginal likelihood hyper-parameter estimation.

Logarithmic inflation of credible sets improves coverage accuracy.

Results extend beyond Gaussian white noise to regression and classification models.

Abstract

We investigate the frequentist coverage properties of credible sets resulting in from Gaussian process priors with squared exponential covariance kernel. First we show that by selecting the scaling hyper-parameter using the maximum marginal likelihood estimator in the (slightly modified) squared exponential covariance kernel the corresponding credible sets will provide overconfident, misleading uncertainty statements for a large, representative subclass of the functional parameters in context of the Gaussian white noise model. Then we show that by either blowing up the credible sets with a logarithmic factor or modifying the maximum marginal likelihood estimator with a logarithmic term one can get reliable uncertainty statement and adaptive size of the credible sets under some additional restriction. Finally we demonstrate on a numerical study that the derived negative and positive results extend beyond the Gaussian white noise model to the nonparametric regression and classification models for small sample sizes as well.

Figures

Tables

Table 1. Table 1. Frequencies that f 2 ​ ( x ) subscript 𝑓 2 𝑥 f_{2}(x) is inside of the corresponding credible interval for the squared exponential Gaussian process prior at given points x ∈ { 0.25 , 0.3188 , 0.75 } 𝑥 0.25 0.3188 0.75 x\in\{0.25,0.3188,0.75\} . Method 1: MMLE empirical Bayes procedure, Method 2: empirical Bayes procedure with log ⁡ n 𝑛 \log n blow up factor, Method 3: modified empirical Bayes procedure (MMLE multiplied by log ⁡ n 𝑛 \log n ). From left to right the sample size is n = 100 , 500 , 1000 𝑛 100 500 1000 n=100,500,1000 .

	$x=0.25$			$x=0.3188$			$x=0.75$
$n=$	$100$	$500$	$1000$	$100$	$500$	$1000$	$100$	$500$	$1000$
Method 1	0.84	0.69	0.57	0.01	0.01	0.00	0.98	0.92	0.97
Method 2	1.00	1.00	1.00	0.96	0.98	1.00	1.00	1.00	1.00
Method 3	0.98	0.98	0.97	0.35	0.55	0.50	0.99	0.96	0.98

Table 2. Table 2. Average size of the pointwise credible intervals (i.e. 2 ​ q 0.025 ​ c ^ ​ ( x , x ) 2 subscript 𝑞 0.025 ^ 𝑐 𝑥 𝑥 2q_{0.025}\sqrt{\hat{c}(x,x)} ) for f 2 ​ ( x ) subscript 𝑓 2 𝑥 f_{2}(x) . Method 1: MMLE empirical Bayes procedure, Method 2: empirical Bayes procedure with log ⁡ n 𝑛 \log n blow up factor, Method 3: modified empirical Bayes procedure (MMLE multiplied by log ⁡ n 𝑛 \log n ). From left to right the sample size is n = 100 , 500 , 1000 𝑛 100 500 1000 n=100,500,1000 .

$n=$	100	500	1000
Method 1	0.3956	0.2367	0.1814
Method 2	1.8218	1.4711	1.2533
Method 3	0.7541	0.5279	0.4262

Table 3. Table 3. Frequencies that f 2 ​ ( x ) subscript 𝑓 2 𝑥 f_{2}(x) is inside of the corresponding credible interval for squared exponential Gaussian process prior at given points x ∈ { 0.25 , 0.3188 , 0.75 } 𝑥 0.25 0.3188 0.75 x\in\{0.25,0.3188,0.75\} . Method 1: MMLE empirical Bayes procedure, Method 2: empirical Bayes procedure with log ⁡ n 𝑛 \log n blow up factor, Method 3: modified empirical Bayes procedure (MMLE multiplied by log ⁡ n 𝑛 \log n ). From left to right the sample size is n = 100 , 200 , 500 𝑛 100 200 500 n=100,200,500 .

	$x=0.25$			$x=0.3188$			$x=0.75$
$n=$	$100$	$200$	$500$	$100$	$200$	$500$	100	200	500
Method 1	0.90	0.90	0.89	0.29	0.16	0.12	0.92	0.88	0.85
Method 2	1.00	1.00	1.00	0.98	0.98	1.00	1.00	1.00	1.00
Method 3	0.91	0.94	0.95	0.42	0.36	0.45	0.94	0.94	0.95

Table 4. Table 4. Average size of the pointwise credible intervals 2 ​ q 0.025 ​ c ^ ​ ( x , x ) 2 subscript 𝑞 0.025 ^ 𝑐 𝑥 𝑥 2q_{0.025}\sqrt{\hat{c}(x,x)} for f 2 subscript 𝑓 2 f_{2} corresponding to the empirical Bayes procedure (first line), to empirical Bayes procedure with log ⁡ n 𝑛 \log n blow up factor (second line) and to the modified empirical Bayes procedure (third line). From left to right the sample size is n = 100 , 200 , 500 𝑛 100 200 500 n=100,200,500 .

	n=100	n=200	n=500
Method 1	3.2672	0.8209	0.3485
Method 2 $\log n$	15.0461	4.3495	2.1661
Method 3	3.6777	1.2675	0.7575

Equations

$$K(s,t)=E[G_s{G}_t]=b\exp \{-a(t-s{)}^2\},s,t\in T,$$

$$\underset{n}{\mathrm{liminf}}\underset{\beta \in B}{\inf }\underset{f\in {\Theta }^{\beta }}{\inf }{P}_f(f\in {\hat{C}}_n)\ge 1-\alpha ,$$

$$\underset{n}{\mathrm{liminf}}\underset{\beta \in B}{\inf }\underset{f\in {\Theta }^{\beta }}{\inf }{P}_f(\Vert f\Vert \le \hat{C}{r}_{\beta })\ge 1-\alpha ,$$

$$Y(t)={\int }_0^t{f}_0(s)ds+\frac{1}{\sqrt{n}}{W}_t,t\in [0,1],$$

$$Y_i=f_{0,i}+\frac{1}{\sqrt{n}}{Z}_i,i=1,2,....$$

$$f_0\in {\Theta }^{\beta }(M)=\{f\in {\ell }_2:\underset{i\ge 1}{\sup }{f}_i^2{i}^{2\beta +1}\le M\},$$

$$\underset{\hat{f}}{\inf }\underset{f\in {\Theta }^{\beta }(M)}{\sup }{E}_0\Vert f-\hat{f}{\Vert }_2\ge C_{\beta }{n}^{-\beta /(1+2\beta )},$$

$$G_t=\sum _{i=1}^{\infty }{\lambda }_i{\xi }_i{\psi }_i(t),$$

$$f|a\sim \underset{i=1}{\overset{\infty }{⨂}}N(0,a^{-1}{e}^{-i/a})$$

$$\displaystyle f|a,Y\sim\bigotimes_{i=1}^{\infty}\mathcal{N}\Bigg{(}\frac{nY_{i}}{ae^{i/a}+n},\frac{1}{ae^{i/a}+n}\Bigg{)}.$$

$${\hat{C}}_{n,\alpha }=\{f\in {\ell }_2:\Vert f-{\hat{f}}_a{\Vert }_2\le r_{\alpha }(a)\}$$

$$\Pi (\cdot )={\int }_0^{\infty }{\Pi }_a(\cdot )\pi (a)da.$$

$${\hat{a}}_n:=\arg \underset{a\in [1,A_n]}{\max }{\ell }_n(a),$$

$$\ell_{n}(a)=-\frac{1}{2}\sum\limits_{i=1}^{\infty}\Bigg{(}\log\Big{(}1+\frac{n}{ae^{i/a}}\Big{)}-\frac{n^{2}Y_{i}^{2}}{ae^{i/a}+n}\Bigg{)}$$

$$c_4^{-1}{a}^{-c_3}\exp (-c_{2}a)\le \pi (a)\le c_4{a}^{-c_5}\exp (-c_{6}a),$$

$${\Pi }_n(f:\Vert f-\hat{f}{\Vert }_2\le r_{\alpha }|Y)=1-\alpha .$$

$${\hat{C}}_n(L_n)=\{f:\Vert f-\hat{f}{\Vert }_2\le L_n{r}_{\alpha }\},$$

$$\displaystyle\sup_{\beta\geq\beta_{0}}\sup\limits_{f\in\Theta^{\beta}(M)}P_{\theta_{0}}\Big{(}{\rm{diam}}(\hat{C}_{n}(1))\geq M_{n}n^{-\beta/(1+2\beta)}(\log n)^{-1/(1+2\beta)}\Big{)}\to 0,$$

$$\underset{f_0\in {\cup }_{\beta \ge {\beta }_0}{\Theta }^{\beta }(M)}{\inf }{P}_0(f_0\in {\hat{C}}_n(L_n))\ge 1-\alpha$$

$${\Theta }_s^{\beta }(m,M)=\{f\in {\Theta }^{\beta }(M):\underset{i\ge 1}{\min }{i}^{1+2\beta }{f}_{0,i}^2\ge m\},$$

$$\underset{f_0\in {\Theta }_s^{\beta }(m,M)}{\sup }{P}_0(f_0\in {\hat{C}}_n(L_n))\to 0.$$

$$f_0\in A^{\gamma }(M)=\{f\in {\ell }_2(M):\sum _{i=1}^{\infty }{f}_i^2{e}^{2i\gamma }\le M\},$$

$$\underset{f_0\in A^{\gamma }(M)}{\inf }{P}_0(f_0\in {\hat{C}}_n(L))\to 1.$$

$$\inf\limits_{f_{0}\in A^{\gamma}(M)}P_{0}\big{(}{\rm{diam}}(\hat{C}_{n}(1))\leq Cn^{-1/2}\log n\big{)}\to 1.$$

$$\sum _{i=N}^{\infty }{f}_i^2\le L_0\sum _{i=N}^{\rho N}{f}_i^2,\text{for all N≥N0}.$$

$$\underset{f_0\in {\Theta }_{pt}(L_0,N_0,\rho )}{\inf }{P}_0(f_0\in {\hat{C}}_n(L\log n))\to 1,$$

$${\tilde{a}}_n=\log (n){\hat{a}}_n.$$

$$\underset{f_0\in {\Theta }_s^{\beta }(m,M)}{\inf }{P}_0(f_0\in {\hat{C}}_n(L))\to 1,$$

$$Y_i=f_0(X_i)+{\epsilon }_i,{\epsilon }_i\stackrel{iid}{\sim }N(0,{\sigma }^2),X_i\stackrel{iid}{\sim }U(0,1),$$

$$P(Y_i=1)=p(X_i),X_i\stackrel{iid}{\sim }U(0,1),$$

$$h_n(a,f_0)$$

Full text

1. Introduction

Bayesian methods are frequently applied in various fields of applications. One, very appealing advantage of the Bayesian framework is that it readily provides built-in uncertainty quantification. In nonparametric problems uncertainty statements are visualized with credible bands, i.e. bands accumulating prescribed fraction (typically $95\%$) of the posterior mass. Gaussian processes are popular and frequently used choices for prior distributions in high-, and infinite dimensional models. Areas of possible applications include machine learning [20], astronomy [10], genomics [15], linguistics [17], epidemiology [2],…etc.

Gaussian processes are characterized by their mean and covariance kernel. Typically one considers centered Gaussian priors. For covariance kernel arguably one of the most frequently used choice is the squared exponential kernel $K(\cdot,\cdot)$, i.e. for centered Gaussian process $G_{t}$,

[TABLE]

for given hyper-parameters $a,b>0$, see for instance [20]. In our paper, we focus on the effect of the hyper-parameter $a$, which has been investigated in the context of various models including nonparametric regression, density estimation, and classification, see for instance [32, 6]. We take the hyper-parameter $b=1$ fixed for simplicity.

In our work we adopt a frequentist perspective, i.e. we assume that the data is generated from some unknown, but fixed probability distribution $P_{f_{0}}$, indexed by a true underlying functional parameter of interest $f_{0}$. We are interested in recovering $f_{0}$ using Bayesian methodology or in other words we are interested in the performance of Bayesian techniques for recovering the underlying functional parameter $f_{0}$. We consider the asymptotic regime assuming that sample size or signal-to-noise ratio increases indefinitely. The frequentist properties of Bayesian methods has been extensively studied in the literature in general high-dimensional and non-parametric settings, see for instance [11, 12, 25]. In these papers it was shown that under relatively mild conditions on the prior and the likelihood function the posterior distribution contracts around the true parameter of interest at (in many instances) optimal rate in various high-dimensional and non-parametric problems. These results show that not only the point estimators (e.g. posterior mean, $\delta$-posterior mode, etc…) resulting from the posterior provide typically good recovery of the truth under the frequentist data generating process, but also the spread of the posterior is not too large and most of the posterior mass is concentrated in a ball centered around the truth with optimal diameter.

In the literature, due to its importance, the posterior contraction rates associated to Gaussian process priors with squared exponential kernel were also separately investigated and it was shown that for appropriate choice of the hyper-parameter $a$, depending on the smoothness of the underlying functional parameter of interest, the posterior distribution achieves nearly the optimal (minimax) contraction rate, see for instance [32, 33, 6, 3]. In practice the regularity of the underlying function is typically not known, therefore one either endows the scaling hyper-parameters with an additional layer of prior distribution (resulting in the so called hierarchical prior distribution, see also [33]) or estimate them from the data (using typically the maximum marginal likelihood estimator). In our work we focus on investigating the reliability of these procedures for uncertainty quantification. Assuming that our observations are generated via a true $f_{0}$, we are interested in whether the credible sets contain this function. In our theoretical analysis we consider the Gaussian white noise model, which is closely related to various nonparametric models and can be thought of as the idealized, continuous observation version of nonparametric regression model.

Although Bayesian methods are routinely used for uncertainty quantification, it is rather unclear whether they can provide reliable uncertainty statements. In fact it is well known that it is impossible to construct confidence sets (either based on purely frequentist or Bayesian methods) which has optimal size over a wide range of regularity classes and provides reliable uncertainty quantification simultaneously. More precisely let us assume that we have a collection of functional classes $\Theta^{\beta}$ indexed by some regularity hyper-parameter $\beta\in B$ and let us denote the minimax estimation rate corresponding to this class by $r_{n,\beta}$ (where $n$ denotes the signal-to-noise ratio). Then in general it is not possible to construct an “honest” confidence set $\hat{C}_{n}$ which achieves simultaneously that

[TABLE]

for some given significance level $\alpha>0$, see for instance [18, 5, 23, 14]. Therefore one has to introduce additional assumptions on the functional parameter $f_{0}$ to obtain confidence sets with optimal size and reliable uncertainty quantification. In the literature various constraints were proposed to overcome this problem, see for instance the monograph [14] for a collection of such approaches.

The coverage properties of credible sets have been investigated only recently, see for instance in [7, 28, 27, 24, 31, 1, 26, 22, 8], and references therein, for various combinations of models and priors. In these papers it was shown that for appropriate choices of the prior distribution both the hierarchical and empirical Bayes procedures can provide in various nonparametric models reliable uncertainty statements under some additional regularity assumption on the underlying functional parameter (e.g. self-similarity assumption, polished tail condition, excessive bias assumption, etc). In our work we focus on the Gaussian process prior with (approximately) squared exponential kernel and show that the empirical Bayes procedure results in unreliable uncertainty statement for a large class of functions. The derived negative theoretical results are also demonstrated via simulation study, and extended to the nonparametric regression and classification models. This troubling finding might shatter our trust in this popular and frequently applied Bayesian technique. However, we propose a simple and intuitive fix for this problem by slightly modifying the maximum likelihood estimator used in the empirical Bayes method. This modification corrects for the haphazard, unreliable uncertainty statements in theory and we demonstrate its applicability in a numerical example for small sample sizes as well.

The paper is organized as follows. In Section 2.1 we introduce the Gaussian white noise model and the considered Bayesian approach in details. Then we present first in Section 2.2 our negative findings on the coverage properties of Bayesian $L_{2}$-credible sets, and then propose different modifications correcting the haphazard behaviour of the posterior by either blowing up the credible sets by a logarithmic factor or (slightly) modifying the marginal maximum likelihood estimator. We show that the proposed methods indeed correct the overconfident uncertainty statements and result in reliable uncertainty quantification for polished tail and self-similar functions, respectively. We demonstrate our findings on a simulation study in Section 3 and discuss the derived results and possible extensions in Section 4. The proofs of the above results are deferred to the Appendix. Finally as a byproduct we also derive contraction rates for the empirical and hierarchical Bayes procedures for a wide range of priors on the rescaling hyper-parameter, extending the results available in the literature. The contraction rate results and their proofs are also deferred to the Appendix.

2. Main results

2.1. Model description

We consider the Gaussian white noise model

[TABLE]

where $f_{0}\in L_{2}[0,1]$ is the unknown function of interest and $W_{t}$ denotes the Brownian motion. Let $P_{0},E_{0}$, and $V_{0}$ denote the corresponding probability measure, expected value, and variance, respectively. This model is closely related to the popular nonparametric regression and density estimation models [19, 4] and can be used as a platform to investigate more complex statistical models, see for instance [29, 14]. In the Bayesian approach we endow the unknown function of interest $f_{0}$ with a prior distribution representing our initial belief. In our work we investigate the popular Gaussian process prior with rescaled squared exponential kernel (1.1). Let us consider the sequence representation of the Gaussian white noise model. For an orthonormal basis $\psi_{i}$, $i=1,2,...$ (e.g. the Fourier basis) let us denote the sequence decomposition of the functions $f_{0}(t),Y(t)$, and $W_{t}$ by $Y_{i}=\langle Y(t),\psi_{i}(t)\rangle_{2}$, $f_{0,i}=\langle f_{0},\psi_{i}(t)\rangle_{2}$, and $Z_{i}=\langle W_{t},\psi_{i}(t)\rangle_{2}\stackrel{{\scriptstyle iid}}{{\sim}}N(0,1)$, $i=1,2,...,$ respectively. Then the equivalent sequence model can be given in the form

[TABLE]

Slightly abusing our notations we denote by $f_{0}$ both the functional parameter in the Gaussian white noise model and the sequential parameter $f_{0}=(f_{0,1},f_{0,2},...)$ in the sequence model. It is common to assume that the true function $f_{0}$ belongs to a hyper-rectangle regularity class, i.e.

[TABLE]

for some (typically unknown) $\beta,M>0$. We note that the minimax estimation rate for the above hyper-rectangle regularity class is $n^{-\beta/(1+2\beta)}$, i.e. there exists $C_{\beta}>0$ such that

[TABLE]

where the infimum is taken over all estimators, see for instance [9].

In view of Mercel’s theorem we can represent the Gaussian process prior with squared exponential kernel as

[TABLE]

where $\lambda_{i}$, $\psi_{i}$, $i=1,2,...$ are the eigenvalues and eigenfunctions of the squared exponential covariance kernel, and $\xi_{i}$ are iid standard normal random variables, see for instance Chapter 4.3 of [21]. The corresponding coefficients $\lambda_{i}$ can be approximated as $\lambda_{i}^{2}\approx a^{-1}e^{-i/a}$, see for instance [21, 3]. In the rest of the paper we work with the prior

[TABLE]

in the sequence model, for convenience. Note that in view of $Y|f_{0}\sim\bigotimes_{i=1}^{\infty}N(f_{0,i},n^{-1})$ and the choice of the prior $\Pi_{a}(\cdot)$ in (2.2), the posterior $\Pi_{a}(\cdot|Y)$ takes the form

[TABLE]

2.2. Uncertainty quantification

In our work we investigate the reliability of the built-in uncertainty quantification of the above posterior distribution. In Bayesian methods the remaining uncertainty of the procedure is visualized by the credible set. We consider $\ell_{2}$-credible balls, i.e. we analyze credible sets in the form

[TABLE]

where $\hat{f}_{a}$ is the posterior mean and the radius $r_{\alpha}(a)$ is chosen such that $\Pi_{a}(f\in\hat{C}_{n,\alpha}|Y)=1-\alpha$, for some prescribed significance level $\alpha>0$ and hyper-parameter $a>0$.

The behaviour of the posterior distribution substantially depends on the choice of the hyper-parameter $a$. For $a=n^{1/(1+2\beta)}$ the corresponding posterior achieves reliable uncertainty quantification in the sense that the credible sets are asymptotically confidence sets, see for instance [32, 3]. In practice, however, the true smoothness parameter $\beta$ is unknown, hence one has to use the data to find the optimal scaling hyper-parameter $a$. The two most commonly applied Bayesian methods for selecting the hyper-parameter are the hierarchical Bayes and the marginal likelihood empirical Bayes methods. In the hierarchical Bayes method the hyper-parameter $a$ is endowed with a prior distribution $\pi$ (also called hyper-prior distribution), resulting in a two-level, hierarchical prior distribution

[TABLE]

In contrast to this in the empirical Bayes approach we take the maximum marginal likelihood estimator (MMLE), i.e.

[TABLE]

where the marginal log-likelihood function (with respect to the measure $\bigotimes_{i=1}^{\infty}N(0,1)$) is

[TABLE]

and the parameter $A_{n}=o(n)$ restricts the parameter space to a compact interval, which is advantageous both from practical and analytical perspective. Then the estimator $\hat{a}_{n}$ is plugged in into the posterior distribution (2.3).

For technical reasons, we introduce the following assumptions on the hyper-prior density function $\pi(\cdot)$ supported on $[1,A_{n}]$.

Assumption 1**.**

Let us assume that for some $c_{1}>0$ there exist $c_{2},c_{6}\geq 0$ and $c_{3},c_{4},c_{5}>0$ such that

[TABLE]

for all $c_{1}\leq a\leq A_{n}$.

Note that amongst others the exponential, the gamma, and the inverse gamma distributions (restricted to $[1,A_{n}]$) satisfy Assumption 1.

We are interested in the frequentist properties of $\ell_{2}$-credible balls resulting from the data driven credible balls. For convenience let $\Pi_{n}(\cdot|Y)$ denote both the hierarchical and the empirical Bayes posterior distributions in the following. Then let us denote by $r_{\alpha}$ the radius of the $\ell_{2}$-ball centered around the posterior mean $\hat{f}$ and accumulating $1-\alpha$ fraction of the posterior mass, i.e.

[TABLE]

In our analysis we introduce some additional flexibility by considering inflated credible balls, i.e.

[TABLE]

for some blown up factor $L_{n}\geq 1$, possibly depending on $n$. As a first step we note that the size of the credible set for both the empirical and hierarchical Bayes procedures adapts to the minimax rate (actually the diameter of the set is even a logarithmic factor faster than the minimax rate).

Corollary 2.1**.**

Both the hierarchical and the empirical Bayes credible sets defined in (2.7) have rate adaptive size, i.e. for every $\beta_{0}>0$ and $M>0$

[TABLE]

where the sequence $M_{n}$ goes to infinity arbitrary slowly in case of the empirical Bayes method and $M_{n}\gg\log n$ in case of the hierarchical Bayes method, and ${\rm{diam}}(S)$ denotes the $\ell_{2}$-diameter of the set $S\subset\ell_{2}(M)$.

Proof.

The proof is given in Sections D and G. ∎

2.2.1. Coverage of credible sets - negative results

Next we investigate how much we can trust the above derived data-driven Bayesian uncertainty quantification from a frequentist perspective. We would like to know whether the true function $f_{0}$ is included in the (blown up) credible set, i.e. if

[TABLE]

holds for some sufficiently large choice of $L_{n}$? Since it is impossible to construct honest confidence sets with rate adaptive size and in view of the adaptive size of the credible sets (see Corollary 2.1), they must have poor frequentist coverage properties at least for certain functional parameters $f_{0}$. Actually the radius of the credible sets are even faster than the minimax rate, which already implies impossibility of coverage. Nevertheless it is of interest to quantify the set of functions for which the Bayesian uncertainty quantification is truth-worthy.

First we note that a representative subset of the hyper-rectangle $\Theta^{\beta}(M)$ is the set

[TABLE]

for some parameters $0<m\leq M$. Let us refer to this subclass of sequential parameters as self-similar signals following the similar terminology of [13, 28]. It was shown in the later paper that the minimax rate over $\Theta^{\beta}_{s}(m,M)$ is the same as over $\Theta^{\beta}(M)$. The next theorem shows that the empirical Bayes procedure provides unreliable uncertainty quantification over this representative sub-class of functions unless it is blown up with at least a logarithmic factor.

Theorem 2.2**.**

Let us take arbitrary $L_{n}=o(\sqrt{\log n})$. Then the empirical Bayes credible set blown up by $L_{n}$ has frequentist coverage tending to zero for every self-similar signal, i.e. for every $0<m\leq M$

[TABLE]

Proof.

See Section D. ∎

This negative result draws a dark picture as it tells us that one can not trust Bayesian uncertainty quantification resulting from the investigated prior, even if one allows certain amount of adjustment (i.e. by blowing up the set with a sequence tending to infinity, not too fast). Since the investigated prior is very closely related to the Gaussian process with squared exponential covariance kernel this gives the intuition that one has to be very cautious working with squared exponential kernel as the corresponding Bayesian uncertainty statement are (typically) unreliable. In the next subsection we will be touching the corners by deriving some positive results on the coverage properties of the credible sets. First we show that for analytic functions the (slightly inflated) credible sets provide reliable uncertainty quantification and second we show that by blowing up the credible sets by a $\log n$ factor or by slightly adjusting the maximum marginal likelihood estimator, one gets reliable uncertainty statements for a large subclass of functions, including the self-similar functions.

2.2.2. Coverage of credible sets - positive results

Let us consider the set of analytic-type functions defined as

[TABLE]

for some $\gamma>0$. Note that the investigated prior (2.2) is more suitable for this class of functions due to the exponential decay of the variances. We show below that, indeed, for the class $A^{\gamma}(M)$ the empirical Bayes procedure provides reliable uncertainty quantification. Note, however, that the present class of functions is substantially smaller than $\Theta^{\beta}(M)$, for any $\beta>0$.

Theorem 2.3**.**

The inflated empirical Bayes credible set $\hat{C}_{n}(L)$ has frequentist coverage tending to one over the class $f_{0}\in A^{\gamma}(M)$ for any $\gamma\geq 1/2$ and sufficiently large constant $L>0$, i.e.

[TABLE]

Furthermore, the size of the credible set is (nearly) optimal, i.e. for some sufficiently large constant $C>0$,

[TABLE]

Proof.

See Section C. ∎

Next we investigate the behaviour of the credible sets by allowing a logarithmic inflating factor. Since the size of the inflated credible sets are still nearly minimax, the credible sets fail to cover all functional parameter $f_{0}$ of interest, in view of the non-existence result of adaptive confidence sets [5, 23]. Therefore we restrict the investigated class of functions to the so called polished tail class, introduced in [28, 24]. We say that a sequential parameter $f\in\ell_{2}(M)$ belongs to the class of polished tail signals denoted by $\Theta_{pt}(L_{0},N_{0},\rho)$, for some $L_{0},\rho,N_{0}>0$ if

[TABLE]

The above assumption basically requires that knowing the sequential parameter $f$ up to a certain coordinate enables us to draw conclusion about the tail of the sequence. We require that the energy (sum of the squared coefficients) of the tail is dominated by the energy of a finitely large block of coefficients. This condition makes also sense intuitively as in the stochastic model the signal can be observed only up to some limit, the fluctuation in the later coordinates can equally likely be caused by the noise. Therefore to make reliable uncertainty statement we have to assume that the tail behaviour of the signal hidden by the noise is not substantial and can be extrapolated by information available at given signal-to-noise ratio. In [28] it was shown that the above assumption is mild from statistical, topological and Bayesian point of view.

The next theorem states that when the sequential parameter $f_{0}$ is restricted to polished tail sequences, then both the empirical and hierarchical Bayes credible balls blown up by a $\log n$ factor (i.e $\hat{C}_{n}(L\log n)$) are honest frequentist confidence set, if $L$ is large enough.

Theorem 2.4**.**

For any $L_{0},N_{0},\rho\geq 1$ there exists a constant $L$ such that

[TABLE]

where $\hat{C_{n}}$ denotes either the empirical or hierarchical Bayes credible sets under Assumption 1.

Proof.

See Sections B and G.2. ∎

Hence one can achieve reliable uncertainty quantification on an arguably large subset of the parameter space by blowing up the standard credible set with a slowly varying term. This, however, is not very appealing as a practitioner would righteously hesitate introducing the artificial logarithmic blow up. Therefore, we propose another method, where one does not have to introduce a logarithmic blow up factor, but instead adjust the maximum marginal likelihood estimator. Investigating the proof of Theorem 2.2 one can see that the MMLE $\hat{a}_{n}$, given in (2.5), is too small, the empirical Bayes procedure is oversmoothing. One can compensate for this by undersmoothing the procedure. We propose to adjust the MMLE by a multiplicative logarithmic factor

[TABLE]

Then the corresponding empirical Bayes credible set (blown up by a sufficiently large constant $L>0$) results in reliable uncertainty quantification for self-similar functions $\Theta^{\beta}_{s}(m,M)$.

Theorem 2.5**.**

For any $0<m\leq M$ there exists a constant $L>0$ such that

[TABLE]

where $\hat{C_{n}}(1)$ denotes the credible set resulting from the empirical Bayes posterior with hyper-parameter $\tilde{a}_{n}$.

The proof of the theorem is deferred to Section F.

3. Simulation study

In this section we investigate the numerical properties of the Gaussian process prior with (approximately) squared exponential covariance kernel. First we consider the Gaussian white noise model and the prior (2.2). We show that the corresponding Bayesian uncertainty quantification is misleading for various regularly behaving functional parameters. We also demonstrate that a different choice of the covariance kernel or a modified version of the empirical Bayes procedure results in more accurate uncertainty statements. Then we consider the (from practical point of view) more relevant nonparametric regression and classification models, where we also demonstrate the sub-optimal behaviour of the (standard) empirical Bayes method with squared exponential covariance kernel and show that the proposed modification results in superior performance compared to it.

3.1. Gaussian white noise model

First we demonstrate the suboptimal performance of the Gaussian process with (approximately) squared exponential covariance kernel (2.2) compared to modified versions of the empirical Bayes procedure and to the Gaussian process prior with polynomially decaying variances in the series representation, see [16, 28]. Let us consider the function $f_{1}\in L_{2}[0,1]$ given by their Fourrier coefficients $f_{1,i}=i^{-3/2}\sin(i)$, for $i=1,2,...$, respectively, relative to the Fourrier eigenbasis $\psi_{i}(t)=\sqrt{2}\cos(\pi(i-1/2)t)$. Note that although the function lies outside of the self-similar function class (2.8), it has essentially the same behaviour. In Figure 1 we visualize the $95\%$ credible sets (light blue or light red), the posterior mean (blue or red) and the true function (black), by simulating 2000 draws from the empirical Bayes posterior distribution and plotting the closest $95\%$ of them in $L_{2}$-norm to the posterior mean. We note that all credible sets were constructed without any inflation factor, i.e. $L_{n}=1$ was taken (except of the case where the choice $L_{n}=\log n$ was pre-specified). The credible sets are drawn for signal-to-noise ratio $n=100$, $500$, $1000$ and $5000$, respectively. We also plot the same credible sets blown-up by a $\log n$ factor, the credible sets obtained by the modified empirical Bayes procedure (where the MMLE of the scaling parameter $a$ was multiplied by $\log n$) and the empirical Bayes credible sets corresponding to the prior $f\sim\otimes_{i=1}^{\infty}N(0,i^{-1-2\alpha})$, with hyper-parameter $\alpha$ estimated by the MMLE. One can see that the standard marginal likelihood empirical Bayes method provides too narrow credible sets failing to cover the underlying true function. Also note that both modifications of the empirical Bayes credible sets provide good coverage, but in contrast to the overly conservative approach of inflating the credible sets with a logarithmic factor the modification of the MMLE results in more informative uncertainty statement (i.e. smaller credible sets).

3.2. Nonparametric regression and classification

In this section we demonstrate on a simulation study that the results derived for the Gaussian white noise model generalize to more complicated statistical models as well. We consider the popular nonparametric regression and classification models specifically. The empirical Bayes posteriors, posterior means and credible sets are computed in both cases using the MatLab package gpml.

In the nonparametric regression model we assume to observe pairs of random variables $(X_{1},Y_{1}),(X_{2},Y_{2}),...(X_{n},Y_{n})$, where

[TABLE]

and the aim is to estimate the unknown nonparametric regression function $f_{0}$. In the Bayesian approach we endow $f_{0}$ with a Gaussian process prior with squared exponential kernel and estimate the tunning parameter using the MMLE.

In this simulation study we take the Fourier coefficients of the underlying true function $f_{2}$ to be $f_{2,i}=i^{-3/2}\cos(i)$, $i=1,2,...$. We take $\sigma^{2}=1/2$, but in the procedure it is considered to be unknown and estimated with the MMLE $\hat{\sigma}^{2}$. We plot the true function (black), the posterior mean (blue), and the posterior pointwise credible intervals (dashed blue) $[\hat{f}(x)-q_{0.025}\sqrt{\hat{c}(x,x)},\hat{f}(x)+q_{0.025}\sqrt{\hat{c}(x,x)}]$, where $\hat{f}$ is the posterior mean, $q_{\alpha}$ the $\alpha$-th quantile of the standard normal distribution and $\hat{c}(.,.)$ the posterior covariance kernel. We consider the MMLE empirical Bayes method with and without the $\log n$ inflation factor for the credible set, the modified empirical Bayes method (where the MMLE was multiplied by $\log n$), and finally the empirical Bayes method for Matérn covariance kernel. We take the sample size to be $n=100,500,1000$ and $2000$. Observe in Figure 2 that the standard MMLE empirical Bayes method provides unreliable uncertainty quantification in certain points, while the two modified squared exponential credible sets and the empirical Bayes credible sets for the Matérn kernel capture the underlying functional parameter of interest better. Also note that by multiplying the MMLE of the scaling parameter by a $\log n$ factor in the squared exponential kernel case we do not get an overly conservative credible set, unlike in the case when the radius is inflated with a logarithmic factor. Finally, we note that the computation time using the squared exponential kernel is much smaller than working with the Matérn kernel.

We also investigate empirically the frequentist coverage probabilities of the pointwise credible sets by repeating the experiment 100 times and reporting the frequency that the function at given points (we consider $x=(0.25,0.3188,0.75)$ with $0.3188=\text{argmax}_{x\in[0,1]}f_{2}(x)$) is included in the credible interval, see Table 1. Moreover, Table 2 shows the average size of the pointwise credible intervals (i.e. $2q_{0.025}\sqrt{\hat{c}(x,x)}$) depending on the sample size $n$ and the procedure used to compute the credible sets. One can observe similar behaviour to what we have described above. We note that we have left out the Matérn kernel from Table 2 due to the overly long computational time.

Next we consider the nonparametric classification problem. Let us assume that we observe the binary random variables $Y_{1},Y_{2},...,Y_{n}\in\{0,1\}$, with

[TABLE]

for some nonparametric function $p(x):\,[0,1]\mapsto[0,1]$. We write $p(x)$ in the form $p(x)=\psi(f(x))$, with $\psi(x)=e^{x}/(1+e^{x})$, for some function $f(x):\,[0,1]\mapsto\mathbb{R}$. In the Bayesian approach we endow the functional parameter $f(x)$ with a Gaussian process prior with squared exponential kernel. We have again excludes the Matérn kernel from the simulations due to the high computational complexity.

We design similar experiments for the nonparametric classification model as for the nonparametric regression model above, with sample sizes $n=100,200,500$ and $1000$ and the same $f_{2}$ as above. We plot the pointwise credible intervals for $f_{2}$ corresponding to the empirical Bayes procedure, with and without a $\log n$ inflation factor, and to the modified empirical Bayes procedure (where the MMLE is multiplied by a $\log n$ factor), see Figure 3. One can observe that the standard MMLE empirical Bayes procedure produces unreliable uncertainty statements, while by blowing up the credible sets with a logarithmic factor we get overly conservative uncertainty quantifications. These problems are resolved by considering the modified empirical Bayes method, which captures the shape of the underlying functional parameter better and provides more reliable uncertainty statements. We also collect the empirical estimation of the frequentist coverage probabilities of the underlying functional parameter $f_{2}(x)$ at points $x=(0.25,0.3188,0.75)$ in Table 3, underlying the conclusions drawn from the figures above. Moreover, Table 4 shows the size of the average credible interval.

4. Discussion

We have shown that the MMLE empirical Bayes method for Gaussian process prior with (a slightly modified version of the) squared exponential covariance kernel produces misleading uncertainty statement in context of the Gaussian white noise model. The derived negative results were demonstrated on a simulation study in context of the Gaussian white noise model and extended to the nonparametric regression and classification models as well. Hence we can conclude that one has to be very cautious when applying empirical Bayes methods with squared exponential Gaussian processes for uncertainty quantification as typically they provide misleading confidence statements, due to over-smoothing behaviour of the MMLE. We note that the bad performance of the prior (2.2) is not due to the factor $a^{-1}$ in the variance, because similar (but easier) computations show that the prior without the $a^{-1}$ factor behaves suboptimally as well. Hence the over-smoothing and as a result the misleading uncertainty statements are due to the exponential decay of the eigenvalues of the prior and can not be corrected by simply multiplying the variance with the scaling parameter $a$.

One can compensate the haphazard uncertainty statements by blowing up the credible sets with a $\log n$ factor, however, this approach is not appealing from a practical perspective, as demonstrated in our simulation study as well. Instead we propose to modify the MMLE by multiplying it with $\log n$ to compensate for the over-smoothing. This procedure is less conservative than the previous one and hence provides more information about the uncertainty of the method. One can also consider different covariance kernels, with polynomially decaying eigenvalues, like the Matérn kernel, however, these procedures can be computationally less appealing.

Appendix A Some properties of the MMLE

A.1. Deterministic bounds

As a first step we provide deterministic bounds for the marginal maximum likelihood estimator $\hat{a}_{n}$ of the rescaling hyper-parameter $a$. Let us introduce the following functions for $a\in[1,\infty)$:

[TABLE]

These functions are derived from the expected value of the score function, see Section E. Then let us define the deterministic bounds $\underline{a}_{n}$ and $\overline{a}_{n}$ for $\hat{a}_{n}$ with the help of the functions $h_{n}$ and $g_{n}$ as

[TABLE]

with some $b,B,K_{0}>0$ to be specified later and $A_{n}=o(n)$ given in (2.5). Then we show that these bounds sandwich $\hat{a}_{n}$ with high probability.

Theorem A.1**.**

The MMLE $\hat{a}_{n}$ satisfies

[TABLE]

for $\underline{a}_{n},\overline{a}_{n}$ defined in (A.3).

Proof.

See Section E. ∎

We also derive an upper bound for $\overline{a}_{n}$ in case the true function belongs to the hyper-rectangle with regularity hyper-parameter $\beta$.

Proposition A.2**.**

For every $\beta\geq\beta_{0}$ and $\gamma>0$ there exist $C_{\beta,b,M},C_{\gamma,b,M}>0$ such that

[TABLE]

Proof.

Let us start with the proof of the first inequality. We show that for any $b>0$ the inequality $h_{n}(a,f_{0})<b$ holds for $a\geq C_{\beta,b,M}n^{1/(1+2\beta)}(\log n)^{-1-1/(1+2\beta)}$. Let us introduce the notation $I_{a}\equiv a\log(n/a)$. Note that by using the inequalities $ae^{i/a}+n\geq n$ and $ae^{i/a}+n\geq ae^{i/a}$, for all $a\geq 1$, and the sum of geometric series we get

[TABLE]

For $A_{n}\geq a\geq C_{\beta,b,M}n^{1/(1+2\beta)}(\log n)^{-1-1/(1+2\beta)}$ the preceding display is bounded by a multiple of $MC_{\beta,b,M}^{-1-2\beta}$. Then for sufficiently large choice of the constant $C_{\beta,b,M}$, we get that $h_{n}(a,f_{0})<b$ for any $a\geq C_{\beta,b,M}n^{1/(1+2\beta)}(\log n)^{-1-1/(1+2\beta)}$.

The proof of the second inequality of the statement goes similarly, i.e. we prove that for $a\geq C_{\gamma,b,M}$ we have $h_{n}(a,f_{0})<b$. Note that by the sum of geometric series we get for every $a\geq 1/\gamma$

[TABLE]

which is bounded from above by arbitrarily small $b$ for sufficiently large choice of the constant $C_{\gamma,b,M}$. ∎

Finally we show that under the polished tail condition the deterministic bounds $\underline{a}_{n},\overline{a}_{n}$ are close to each other.

Lemma A.3**.**

For every $L_{0},\rho,N_{0}\geq 1$ we have

[TABLE]

with $K=8.1e^{4}\rho^{2}L_{0}B/b$ for $n$ large enough.

Proof.

First of all note that since $\underline{a}_{n}\leq\overline{a}_{n}$, there is nothing to prove in the trivial cases $\underline{a}_{n}=A_{n}$ or $\overline{a}_{n}=K_{0}$. Hence $h_{n}(\overline{a}_{n},f_{0})\leq b$ and $g_{n}(a,f_{0})<B\log n$, for all $a>\underline{a}_{n}$, hold. Furthermore assume that $\underline{a}_{n}\leq\rho^{-2}\overline{a}_{n}$, else the statement is trivial.

Let us divide the interval $[\rho^{j},\rho^{j+1})$ into sub-intervals $[\rho^{j+k/\lceil\log n\rceil},\rho^{j+(k+1)/\lceil\log n\rceil})$, $k=0,1,...\lceil\log n\rceil-1$, and introduce the notation

[TABLE]

with the notational convenience $\sum_{i=a}^{b}c_{i}=\sum_{i=\lceil a\rceil}^{\lfloor b\rfloor}c_{i}$, applied later on as well.

Then by the polished tail condition

[TABLE]

for $j\geq\log_{\rho}N_{0}$. Note that for every $a>0$ there exists an $\tilde{a}\in(a,\rho^{2}a)$ such that

[TABLE]

for some $j\in\mathbb{N}$ and let us denote this $j$ by $J_{\tilde{a}}$. Then

[TABLE]

Let us take any $a_{1}\leq\rho^{-2}a_{2}$ and denote by $\tilde{a}_{1}\in(a_{1},\rho^{2}a_{1})$ the value satisfying (A.5). Then by the previous inequalities and noting that $\exp\{e^{1/\log n}\log(n/a)\}\leq\exp\{(1+2/\log n)\log(n/a)\}\leq e^{2}n/a$, for $n\geq e$, using that $e^{x}\leq 1+2x$ for $x\in[0,1]$,

[TABLE]

Since $ie^{i/\tilde{a}_{1}}>e^{-2}n\log(n/\tilde{a}_{1})$ for $i\geq e^{-1/\log n}I_{\tilde{a}_{1}}$, and $ie^{i/a_{2}}\leq n\log(n/a_{2})$ for $i\leq I_{a_{2}}$, we can see that

[TABLE]

Moreover, since

[TABLE]

combined with the preceding computations we get that

[TABLE]

Furthermore, let us note that for any $\underline{a}_{n}<a\leq A_{n}$

[TABLE]

Then by taking $a_{1}=\underline{a}_{n}$, $\tilde{a}_{1}\in(\underline{a}_{n},\rho^{2}\underline{a}_{n})$, and $a_{2}=\overline{a}_{n}$ in $\eqref{eq: help:dist:bounds}$ we get that

[TABLE]

After rearranging the preceding inequality we arrive to our statement. ∎

A.2. Contraction rates

In this section we provide the contraction rate results both for the empirical and hierarchical Bayes procedures.First we show that the empirical Bayes method achieves the (up to a logarithmic factor) optimal minimax contraction rate around the truth for unknown regularity hyper-parameter $\beta>0$.

Theorem A.4**.**

The maximum marginal likelihood empirical Bayes posterior corresponding to the prior (2.2) achieves the minimax adaptive contraction rate (up to a logarithmic factor), i.e. for given $M,\beta_{0}>0$ we have

[TABLE]

for any sequence $M_{n}$ tending to infinity.

Proof.

See Section A.3. ∎

Using our findings on the empirical Bayes method we can extend the results on the hierarchical Bayes method, derived in the literature (where typically inverse gamma hyper-prior was considered), by allowing other, more general choices of the hyper-prior distribution as well.

Theorem A.5**.**

Let us assume that the hyper-prior $\pi$ satisfies Assumption 1. Then the corresponding hierarchical Bayes posterior achieves the minimax contraction rate (up to a logarithmic factor), i.e. for given $\beta_{0},M>0$ we have

[TABLE]

for some arbitrary sequence $M_{n}$ tending to infinity.

Proof.

See Section G.1. ∎

The above results are of interest in their own right, but our main focus lies on the reliability of Bayesian uncertainty quantification resulting both from the hierarchical and the empirical Bayes procedures, hence we have deferred the contraction rate results to the appendix.

A.3. Proof of Theorem A.4

Let us introduce the shorthand notation $\varepsilon_{n}:=n^{-\beta/(1+2\beta)}(\log n)^{2\beta/(1+2\beta)}$. In view of Markov’s inequality and Theorem A.1, for every $\beta>0$

[TABLE]

where

[TABLE]

is the posterior risk. We show below that both

[TABLE]

hold, which results in that the right-handside of (A.9) vanishes as $n\to\infty$, concluding the proof of Theorem A.4.

A.3.1. Bound for the expected posterior risk (A.10)

First, note that by elementary computations

[TABLE]

where $\hat{f}_{a,i}=n(ae^{i/a}+n)^{-1}Y_{i}$ is the $i$th coefficient of the posterior mean. Therefore the expectation of $R_{n}(a)$ is given by

[TABLE]

Note that the second and third terms do not contain $f_{0}$, and that the second term is bounded by the third. By Lemma H.2 (with $r=0$ and $l=1$) and Proposition A.2 the latter is further bounded for $a\leq\overline{a}_{n}$ by a multiple of

[TABLE]

since the function $a\mapsto a\log(n/a)$ is monotone increasing for $a\leq n/e$. It remained to deal with the first term on the right hand side of (A.12), which we divide into three parts and show that each of the parts have the stated order. First note that for $f_{0}\in\Theta^{\beta}(M)$

[TABLE]

Next note that for $a\leq\overline{a}_{n}$, in view of Proposition A.2,

[TABLE]

Furthermore, notice that the maximum of the function $i\mapsto e^{i/a}/(i-a)$ over $[2a,I_{a}]$ is attained at $i=I_{a}$, because the function is increasing for $i>2a$ and $n>0$. Besides, for $a>\underline{a}_{n}$ we have $g_{n}(a,f_{0})<B\log n$, hence for any $\underline{a}_{n}<a\leq\bar{a}_{n}$

[TABLE]

where the last inequality follows from Proposition A.2.

It remained to deal with the terms between $I_{\underline{a}_{n}}=\underline{a}_{n}\log(n/\underline{a}_{n})$ and $(n/\log^{2}n)^{1/(1+2\beta)}$. Let $J=J(n)$ be the smallest integer such that

[TABLE]

and let

[TABLE]

Note that the sequence $n_{j}$ is increasing. For notational convenience, we also introduce $b_{j}$ such that $b_{j}e^{n_{j}/b_{j}}=n$ and $b_{j}<n_{j}$. Now we have for any $a\geq 1$

[TABLE]

By elementary computations we get that $b_{j}\asymp n_{j}/\log n_{j}$, therefore $\eqref{eq: risk:help1}$ is further bounded by constant times

[TABLE]

Since $b_{j}\geq\underline{a}_{n}$ we have $g_{n}(b_{j},f_{0})\leq B\log n$ for all $j\geq 0$. Then by the sum of geometric series we get that

[TABLE]

concluding the proof of assertion (A.10).

A.3.2. Bound for the centered posterior risk (A.11)

Note that

[TABLE]

Therefore it is sufficient to show that

[TABLE]

We deal with the two processes above, separately.

For the process $\mathbb{V}$, Corollary 2.2.5 in [34] implies that

[TABLE]

where $d_{n}^{2}(a_{1},a_{2})=V_{0}(\mathbb{V}(a_{1})-\mathbb{V}(a_{2}))$, $diam_{n}$ is the $d_{n}$-diameter of $[\underline{a}_{n},\overline{a}_{n}]$ and $N(\varepsilon,B,d_{n})$ the covering number of the set $B$ with $\varepsilon$-radius balls relative to the $d_{n}$ semi-metric. The variance of $\mathbb{V}(a)$ is equal to

[TABLE]

since $V(Z_{i}^{2})=2$. Using Lemma H.2 (with $r=0$ and $l=4$) we can conclude that the variance of $\mathbb{V}(a)$ is bounded from above by a multiple of $a\log(n/a)$, hence $diam_{n}\lesssim\sqrt{\overline{a}_{n}\log n}$. In view of Lemma A.6, the distance $d_{n}(a_{1},a_{2})$ is bounded from above by a multiple of $|a_{1}-a_{2}|\log^{3/2}n$, hence the interval $[\underline{a}_{n},\overline{a}_{n}]$ can be covered with constant times $\overline{a}_{n}\varepsilon^{-1}\log^{3/2}n$ amount of $\varepsilon$-balls relative to the $d_{n}$ semi-metric. In view of the above computation and Proposition A.2

[TABLE]

The process $\mathbb{W}$ can be dealt with similarly to $\mathbb{V}$. The main difference is the bounding of the variance of $\mathbb{W}$, which we describe in details. First note that

[TABLE]

Let us split the sum at $I_{a}$ and by applying the inequality $ae^{i/a}+n\geq n$, we get

[TABLE]

Then by noting that the function $i\mapsto e^{i/a}/((i-a)(ae^{i/a}+n)^{2})$ is decreasing on $[I_{a},\infty)$, recalling that $g_{n}(a,f_{0})\leq B\log n$, for all $a\geq\underline{a}_{n}$, and in view of Proposition A.2

[TABLE]

hence ${\rm{diam}}_{n}=O(n^{-\frac{1/2+2\beta}{1+2\beta}}(\log n)^{\frac{\beta}{1+2\beta}}))$. Then in view of Lemma A.6 the covering number of the interval $[\underline{a}_{n},\overline{a}_{n}]$ is bounded by $\varepsilon^{-1}(\overline{a}_{n}/\sqrt{n})\log n$ with respect to the semi-metric d_{n}(a_{1},a_{2})=V_{0}\big{(}\mathbb{W}(a_{1})/\sqrt{n}-\mathbb{W}(a_{2})/\sqrt{n}\big{)} and the rest of the proof goes as above.

A.3.3. Bounds for the semimetrics associated to $\mathbb{V}$ and $\mathbb{W}$

Lemma A.6**.**

For any $1\leq a_{1}\leq a_{2}$ and $f_{0}\in\ell_{2}(M)$ we have

[TABLE]

with constants only depending on $M$.

Proof.

The left-hand side of the first inequality is equal to

[TABLE]

where $\phi_{i}(a)=(ae^{i/a}+n)^{-2}$. The square of the derivative of $\phi_{i}$ is given by $\phi_{i}^{\prime}(a)^{2}=4\phi_{i}(a)^{3}e^{2i/a}(i-a)^{2}/a^{2}$, hence in view of Lemma H.3 the preceding display is bounded above by a multiple of

[TABLE]

with the help of Lemma H.1 (first with $m=2$ and then with $m=0$), and Lemma H.2 (with $r=1$ and $l=4$).

We next consider the process $\mathbb{W}(a)$. The left-hand side of the second inequality in the statement of the lemma is equal to

[TABLE]

with $\phi_{i}(a)=ae^{i/a}/(ae^{i/a}+n)^{2}$. Note that $|\phi_{i}^{\prime}(a)|\leq(i+a)a^{-2}\phi_{i}(a)$, hence in view of Lemma H.1 (first with $m=2$ and then with $m=0$) and Lemma H.3 the preceding display is bounded by

[TABLE]

concluding the proof of the lemma. ∎

Appendix B Proof of the empirical Bayes part of Theorem 2.4

First note that we get the empirical Bayes credible set by plugging in the estimator $\hat{a}_{n}$ into the credible set $\hat{C}(a)$ with fixed hyper-parameter $a$, given in (2.4). The proof of the statement is then based on the deterministic bounds for the MMLE $\hat{a}_{n}$ derived in Theorem A.1 and their distance investigated in Lemma A.3.

Note that $f_{0}\in\hat{C}_{n}(L\log n)$ if and only if $\|f_{0}-\hat{f}\|_{2}\leq L\log(n)r_{\alpha}$. Therefore by triangle inequality it is sufficient to verify that

[TABLE]

holds with high probability, where $W(a)=\hat{f}_{a}-E_{0}\hat{f}_{a}$ and $B(a,f_{0})=E_{0}\hat{f}_{a}-f_{0}$ are the centered posterior mean and the bias of the posterior mean for fixed hyperparameter $a>0$, respectively. Note that the $i$th coefficient of these vectors take the form

[TABLE]

We prove below that there exist constants $C_{1},C_{2},C_{3}>0$ such that

[TABLE]

Hence in view of Theorem A.1 assertion $\eqref{eq: help_coverage}$ holds with probability tending to one for large enough choice of $L$, under the polished tail assumption.

Proof of (B.2): The radius $r_{\alpha}(a)$, given in (2.4), is defined as $P(U_{n}(a)<r_{\alpha}^{2}(a))=1-\alpha$ with $U_{n}(a):=\sum\limits_{i=1}^{\infty}\frac{1}{ae^{i/a}+n}Z_{i}^{2}$, where $Z_{i}$’s are iid $N(0,1)$. We show below that

[TABLE]

Then by Markov’s inequality with probability tending to one we have

[TABLE]

hence (B.2) follows from the definition of $r_{\alpha}(a)$.

Assertion (B.5) follows as

[TABLE]

To verify (B.6), it suffices by Corollary 2.2.5 in [34] (applied with $\psi(x)=x^{2}$) to show that there exist $K_{1},K_{2}>0$ such that for any $a\in[\underline{a}_{n},\overline{a}_{n}]$

[TABLE]

where $d_{n}$ is the semimetric defined by d_{n}^{2}(a_{1},a_{2}):=V\Big{(}\frac{nU_{n}(a_{1})}{a_{1}\log(n/a_{1})}-\frac{nU_{n}(a_{2})}{a_{2}\log(n/a_{2})}\Big{)}, $diam_{n}$ is the diameter of the interval $[\underline{a}_{n},\overline{a}_{n}]$ relative to $d_{n}$ and $N(\varepsilon,S,d_{n})$ is the minimal number of $d_{n}$-balls of radius $\varepsilon$ needed to cover the set $S$.

First note that in view of Lemma H.2 (with $r=0$ and $l=2$) we have

[TABLE]

As a consequence one can see that $diam_{n}\lesssim(\underline{a}_{n}\log(n/\underline{a}_{n}))^{-1/2}$. By Lemma B.1, $d_{n}(a_{1},a_{2})\lesssim a_{1}^{-3/2}\log^{1/2}(n/a_{1})n^{-1}|a_{1}-a_{2}|$, hence

[TABLE]

Therefore one can conclude that

[TABLE]

Proof of (B.3): The variable $\|W(a)\|^{2}_{2}$ is distributed as $\sum\limits_{i=1}^{\infty}\frac{n}{(ae^{i/a}+n)^{2}}Z_{i}^{2}$, with $Z_{i}\stackrel{{\scriptstyle iid}}{{\sim}}N(0,1)$. Observe that

[TABLE]

Furthermore note that by applying Lemma H.2 (with $r=0$ and $l=2$) we get

[TABLE]

while by applying the same lemma (with $r=0$ and $l=4$) the variance is bounded above by a multiple of $an^{-2}\log(n/a)$. Then similar reasoning to the previous proof results in that

[TABLE]

Then in view of Lemma A.3, the right hand side of the inequality in the preceding probability statement is further bounded from above by constant times $(\underline{a}_{n}/n)\log(n/\underline{a}_{n})\log^{2}n$.

Proof of (B.4): First note that

[TABLE]

To bound the first term on the right hand side, we use the inequalities $a/n\leq\log(n/a)$ for $a\leq A_{n}$ and $\sum_{i=1}^{\infty}f_{0,i}^{2}<\infty$, and furthermore note the function $i\mapsto e^{i/a}/(i-a)$ is monotone increasing on the interval $[2a,I_{a}]$ hence it takes its maximum at $I_{a}$. Therefore the first part of the bias is bounded by

[TABLE]

using the definition of $\underline{a}_{n}$. Furthermore in view of the polished tail assumption we have

[TABLE]

for some $\tilde{a}_{n}\in[\underline{a}_{n},\rho\underline{a}_{n}]$. Therefore

[TABLE]

Combining the two bounds, we see that (B.4) holds.

Lemma B.1**.**

There exists a $K>0$ such that for any $1<a_{1}<a_{2}$

[TABLE]

Proof.

First note that

[TABLE]

with $\phi_{i}(a):=\frac{1}{a\log(n/a)(ae^{i/a}+n)}$. The derivative of $\phi_{i}(a)$ is given as \phi^{\prime}_{i}(a)=\phi_{i}(a)\Big{(}\frac{2(i-a)e^{i/a}}{a(ae^{i/a}+n)}+\frac{1}{a\log(n/a)}-\frac{1}{a}\Big{)}, so we can see that |\phi^{\prime}_{i}(a)|\lesssim\big{(}\frac{(i+a)e^{i/a}}{a(ae^{i/a}+n)}\vee\frac{1}{a}\big{)}\phi_{i}(a). Thus in view of Lemma H.3 the right hand side of (B.11) is bounded by a multiple of

[TABLE]

Then in view of Lemma H.1 (first with $m=2$ and then with $m=0$) and Lemma H.2 (first with $r=1$ and $l=2$ and second with $r=0$ and $l=2$) the preceding display is further bounded by the right hand side of (B.10), finishing the proof of the statement. ∎

Appendix C Proof of Theorem 2.3

We use the notations introduced in Section B.

First recall that $f_{0}\in\hat{C}_{n}(L)$ if and only if $\|f_{0}-\hat{f}\|_{2}\leq Lr_{\alpha}(\hat{a}_{n})$. We show below that

[TABLE]

which together with (B.2) and Theorem A.1 results in the statement.

The proof of assertion (C.1) follows by combining (B.9) and the second inequality of Proposition A.2. Next note that similarly to the proof of (B.4), we get that

[TABLE]

Furthermore

[TABLE]

for $\gamma\geq 1/2$, finishing the proof of (C.2) and concluding the proof of the theorem.

Appendix D Proof of Theorem 2.2 and the empirical Bayes part of Corollary 2.1

In the proof we use again the notations introduced in Section B.

First note that $f_{0}\in\hat{C}_{n}(L_{n})$ implies that $\|B(\hat{a}_{n},f_{0})\|_{2}\leq L_{n}r_{\alpha}(\hat{a}_{n})+\|W(\hat{a}_{n})\|_{2}$, which combined with Theorem A.1 provides the upper bound

[TABLE]

The proof of assertion (B.2) also shows that there exists constants $C_{1}>0$ such that

[TABLE]

Then in view of assertion (B.9) and Proposition A.2, both the squared radius $r_{\alpha}(a)^{2}$ and the variance term $\|W(a)\|_{2}^{2}$ are bounded by a constant multiplier of $n^{-2\beta/(1+2\beta)}(\log n)^{-1/(1+2\beta)}$.

Since for $f_{0}\in\Theta^{\beta}_{s}(m,M)$ we have $\|B(a,f_{0})\|_{2}^{2}=\sum\limits_{i=1}^{\infty}\frac{a^{2}e^{2i/a}f_{0,i}^{2}}{(ae^{i/a}+n)^{2}}$ the bias is bounded from below by

[TABLE]

As the function a\mapsto a^{-2\beta}\log^{-2\beta}\big{(}n/a\big{)} is monotone decreasing for $a\leq A_{n}$, we see that $\inf\limits_{a\leq\overline{a}_{n}}\|B(a,f_{0})\|_{2}^{2}\gtrsim m\overline{a}_{n}^{-2\beta}\log^{-2\beta}(n/\overline{a}_{n})$. Hence in view of Proposition A.2 the bias is bounded from below by $n^{-2\beta/(1+2\beta)}\log^{2\beta/(1+2\beta)}n$. Thus, the above inequalities imply that for arbitrary $f_{0}\in\Theta^{\beta}_{s}(m,M)$ the right hand side of (D.1) is further bounded by

[TABLE]

for arbitrary $L_{n}=o(\sqrt{\log n})$ concluding the proof of the theorem.

Appendix E Proof of Theorem A.1

First note that the derivative of the marginal likelihood function $\ell_{n}(a)$ is

[TABLE]

with expected value

[TABLE]

In the following subsections we show with the help of the score function $\mathbb{M}_{n}(a)$ that the marginal likelihood function $\ell_{n}(a)$ with probability tending to one has its global maximum outside of the set $[1,\underline{a}_{n})\cup(\overline{a}_{n},A_{n}]$.

E.1. $\mathbb{M}_{n}(a)$ on $[1,\underline{a}_{n})$

In this subsection we derive that the process $\mathbb{M}_{n}(a)$ is bounded from below by $-C\log^{2}(n/a)$ on $[1,\underline{a}_{n}]$, for some $C>0$, and is bigger than $\tilde{c}_{0}B\log^{3}(n/\underline{a}_{n})$, for some $0<\tilde{c}_{0}<\exp\{-2(4+1/\beta_{0})\}/2$, on the interval

[TABLE]

with probability going to one, where $B$ is the parameter in the definition of $\underline{a}_{n}$. Hence with probability tending to one for every $a\in[1,\underline{a}_{n}]/\mathcal{I}_{n}$

[TABLE]

for $B>2\tilde{c}_{0}^{-1}C$. Therefore the global maximum of $\ell_{n}(a)$ lies outside of the interval $[1,\underline{a}_{n})$ with probability tending to one. It remained to show the stated lower bounds for $\mathbb{M}_{n}(a)$.

By leaving the non-negative stochastic part out we get the lower bound

[TABLE]

In view of Lemma E.1 the deterministic part in (E.4) is bounded from below by a negative constant times \log^{2}\big{(}n/a\big{)}. The stochastic part is bounded from below by $-C\sum\limits_{i=1}^{a}Y_{i}^{2}$ and since $E_{0}[\sum\limits_{i=1}^{a}Y_{i}^{2}]=\sum\limits_{i=1}^{a}f_{0,i}^{2}+an^{-1}$ and V_{0}\big{(}\sum\limits_{i=1}^{a}Y_{i}^{2}\big{)}=2n^{-1}\sum\limits_{i=1}^{a}f_{0,i}^{2}+an^{-2}\to 0 for all $a\leq A_{n}$ it follows from Chebyshev’s inequality that the sum $\sum\limits_{i=1}^{a}Y_{i}^{2}$ is bounded with probability going to 1, for all $f_{0}\in\ell_{2}(M)$.

Next we deal with the lower bound on the interval $a\in\mathcal{I}_{n}$. First note that $Y_{i}^{2}\geq f_{0,i}^{2}+2f_{0,i}Z_{i}/\sqrt{n}$ implying

[TABLE]

with the centered Gaussian process

[TABLE]

Note that

[TABLE]

hence the diameter of the interval $\mathcal{I}_{n}$ with respect to the metric

[TABLE]

is bounded by a multiple of $\sup_{a\in\mathcal{I}_{n}}g_{n}(a,f_{0})^{1/2}(a\log(n/a))^{-1/2}$.

Next we give an upper bound for the covering number of the interval $\mathcal{I}_{n}$. Let us take $\epsilon$-balls centered at $a\in\mathcal{I}_{n}$, with $2a\in\mathbb{N}$. To cover the remaining part of the interval $\mathcal{I}_{n}$ it is sufficient to cover all intervals of the form $(a,a+1/2)$, $2a\in\mathbb{N}\cap 2\mathcal{I}_{n}$. Note that on these intervals for every $a_{1},a_{2}\in(a,a+1/2)$ we have $\lfloor 2a_{1}\rfloor-\lfloor 2a_{2}\rfloor=0$. Hence in view of Lemma E.2 we have $d_{n}(a_{1},a_{2})\lesssim|a_{1}-a_{2}|\sup_{a\in\mathcal{I}_{n}}\sqrt{\log(n/a)g_{n}(a,f_{0})/a^{3}}$. Thus the covering number of the interval $(a,a+1/2)$ relative to $d_{n}$ is bounded from above by a multiple of $\varepsilon^{-1}\sup_{a\in\mathcal{I}_{n}}\sqrt{\log(n/a)g_{n}(a,f_{0})/a^{3}}$, which implies that the covering number of the whole interval $\mathcal{I}_{n}$ is bounded from above by constant times $\varepsilon^{-1}\sup_{a\in\mathcal{I}_{n}}\sqrt{\log^{-1}(n/a)g_{n}(a,f_{0})/a}+\underline{a}_{n}/\log(n/\underline{a}_{n})$.

We show below that for any $c_{0}>2$

[TABLE]

hold. Therefore the covering number of $\mathcal{I}_{n}$ is bounded from above by a multiple of $\underline{a}_{n}+\varepsilon^{-1}\sqrt{\log^{-1}(n/\underline{a}_{n})\log(n)/\underline{a}_{n}}$.

By Corollary 2.2.5 in [34] (applied with $\psi(x)=e^{x^{2}}-1$) it follows that

[TABLE]

Therefore the process $\mathbb{M}_{n}(a)$ can be bounded from below on $a\in\mathcal{I}_{n}$ by

[TABLE]

with probability going to one. In view of (E.6) and since the third and fourth terms on the right hand side of the preceding display are bounded from below by a fixed negative constant, we get that with probability tending to one $\mathbb{M}_{n}(a)\geq\tilde{c}_{0}B\log n$, with $\tilde{c}_{0}=e^{-2c_{0}}/2$ .

It remained to verify assertion (E.6). First note that

[TABLE]

Furthermore, in view of the inequality $c_{0}I_{\underline{a}_{n}}(a^{-1}-\underline{a}_{n}^{-1})\leq c_{0}$, for $a\in\mathcal{I}_{n}$, we have that

[TABLE]

By combining the preceding two displays we get that

[TABLE]

finishing the proof of the first inequality in (E.6). The proof of the second inequality goes accordingly.

Lemma E.1**.**

There exists a constant $K>0$ such that for any $a\in[1,n)$

[TABLE]

Proof.

Note that

[TABLE]

∎

Lemma E.2**.**

*There exists a constant $K>0$ such that for any $0<a_{1}<a_{2}$, $\lfloor 2a_{2}\rfloor-\lfloor 2a_{1}\rfloor=0$

[TABLE]

Proof.

Recall that the left hand side of the display in the lemma was denoted by $d_{n}^{2}(a_{1},a_{2})$ and note that

[TABLE]

with $\phi_{i}(a):=\frac{(i-a)e^{i/a}}{\log(n/a)^{2}a(ae^{i/a}+n)^{2}}$. Then by elementary, but cumbersome computations we get that $|\phi^{\prime}_{i}(a)|\lesssim ia^{-2}\phi_{i}(a)$. Thus, in view of Lemma H.3, the right hand side of (E.7) is bounded by

[TABLE]

Then the statement of the lemma follows by applying Lemma H.1 (with $m=3$). ∎

E.2. $\mathbb{M}_{n}(a)$ on $[\overline{a}_{n},A_{n}]$

By assuming $\overline{a}_{n}>K_{0}$, we have $h_{n}(a,f_{0})\leq b$ for $a\in[\overline{a}_{n},A_{n}]$. Next we prove that for sufficiently large choice of $K_{0}>0$

[TABLE]

These imply that with probability tending to one $\mathbb{M}_{n}(a)<-2^{-6}\log^{2}(n/a)$, for every $a\in[\bar{a}_{n},A_{n}]$, hence the marginal likelihood function $\ell_{n}(a)$ is monotone decreasing and does not attain its global (or local) maximum on the interval $[\overline{a}_{n},A_{n}]$, i.e.

[TABLE]

Proof of assertion (E.8): For $a\geq\overline{a}_{n}$ the expectation satisfies

[TABLE]

In view of Lemma H.2 (with $r=0$ and $l=2$), we have $\sum\limits_{i=1}^{\infty}\frac{n^{2}}{a(ae^{i/a}+n)^{2}}\lesssim\log(n/a)$. Furthermore,

[TABLE]

which implies that

[TABLE]

concluding the proof of assertion (E.8), for small enough choice of $b$ ($b<2^{-4}$ is small enough).

Proof of assertion (E.9): In view of Corollary 2.2.5 in [34] (applied with $\psi(x)=x^{2}$) it is sufficient to show that there exist universal constants $K_{1},K_{2}>0$ such that for any $a\in[\overline{a}_{n},A_{n}]$

[TABLE]

where $d_{n}$ is the semimetric defined by d_{n}^{2}(a_{1},a_{2}):=V_{0}\Big{(}\frac{\mathbb{M}_{n}(a_{1})}{\log^{2}(n/a_{1})}-\frac{\mathbb{M}_{n}(a_{2})}{\log^{2}(n/a_{2})}\Big{)}, $diam_{n}$ is the diameter of $[\overline{a}_{n},A_{n}]$ relative to $d_{n}$ and $N(\varepsilon,S,d_{n})$ is the minimal number of $d_{n}$-balls of radius $\varepsilon$ needed to cover the set $S$, since by sufficiently large choice of $K_{0}$ ($K_{0}\geq(2^{6}K_{2})^{4}$ is sufficiently large) assertion (E.9) holds.

Note that Lemma E.3 immediately implies assertion (E.11) and

[TABLE]

Then let us introduce the cover

[TABLE]

with $K_{n}=\lceil\log(A_{n}/\overline{a}_{n})\rceil$. In view of Lemma E.4 on the interval $a_{1},a_{2}\in[2^{k}\overline{a}_{n},2^{k+1}\overline{a}_{n}]$

[TABLE]

hence

[TABLE]

This results in

[TABLE]

Lemma E.3**.**

For all $a\in[\overline{a}_{n},A_{n}]$, we have V_{0}\big{(}\mathbb{M}_{n}(a)/\log^{2}(n/a)\big{)}\lesssim(a\log(n/a))^{-1}.

Proof.

We know that the $Y_{i}$’s are independent and $V_{0}(Y_{i}^{2})=2/n^{2}+4f_{0,i}^{2}/n$, so the variance is equal to

[TABLE]

In view of $(i-a)^{2}\leq a^{2}+i^{2}$, for any $a,i>0$, and by applying Lemma H.1 (with $m=2$) and Lemma H.2 (first with $r=2$ and $l=4$ and then with $r=1$ and $l=2$) the first sum in $\eqref{eq:help:Lem4}$ is bounded from above by a multiple of

[TABLE]

Similarly, following from Lemma H.1 (with $m=1$ and $m=-1$) and $h_{n}(a)\leq b$ for $a\geq\overline{a}_{n}$, the second sum in (E.13) is bounded by a multiple of

[TABLE]

concluding the proof of the lemma. ∎

Lemma E.4**.**

For all $1\leq a_{1}<a_{2}<A_{n}$, we have

[TABLE]

for some universal constant $C_{0}>0$.

Proof.

Note that

[TABLE]

with $\phi_{i}(a)=\frac{e^{i/a}(i-a)}{2a\log^{2}(n/a)(ae^{i/a}+n)^{2}}$. By elementary computations one can see that $|\phi_{i}(a)^{\prime}|^{2}\lesssim(i^{2}a^{-4}+a^{-2})\phi_{i}^{2}(a)$, hence in view of Lemma H.3,

[TABLE]

Since $V_{0}(Y_{i}^{2})=2/n^{2}+4f_{0,i}^{2}/n$ the preceding sum is bounded by

[TABLE]

Then in view of Lemma H.1 (applied with $m=4$ and $m=0$) and Lemma H.2 (applied with $r=1$ and $l=2$) the first term of (E.14) is bounded from above by a multiple of

[TABLE]

Similarly in view of Lemma H.1 (with $m=3$ and $m=-1$) the second term of (E.14) is bounded by

[TABLE]

concluding the proof of the lemma. ∎

Appendix F Proof of Theorem 2.5

Similarly to the previous sections we use the notations introduced in Section B. We show below that there exists a constant $c>0$ depending only on $m,M$ and $\beta_{0}$ such that

[TABLE]

which combined with Proposition A.2 and Theorem A.1 results in

[TABLE]

for some positive constants $c,C$. Let us introduce then the notation

[TABLE]

As before, note that $f_{0}\in\hat{C}_{n}(L)$ is equivalent to $\|f_{0}-\hat{f}\|_{2}\leq Lr_{\alpha}(\tilde{a}_{n})$, hence by proving that

[TABLE]

hold for some constants $C_{1},C_{2},C_{3}>0$, the statement of the theorem follows immediately. The proof of the first two inequalities follow from (B.2) and (B.9) (with $\underline{a}_{n}$ and $\overline{a}_{n}$ replaced by a multiple of $(n/\log n)^{1/(1+2\beta)}$), respectively. To prove the last inequality we note that for $f_{0}\in\Theta_{s}^{\beta}(m,M)$, $a\in\tilde{\mathcal{I}}_{n}$, and $\beta\geq\beta_{0}$ we have that

[TABLE]

It remained to prove assertion (F.1). Let us introduce the slightly modified version of $\underline{a}_{n}$ as

[TABLE]

for some sufficiently large constant $B>0$ to be specified later. Then we show below that

[TABLE]

for some sufficiently small constant $c>0$.

For the second statement note that

[TABLE]

hence for any fixed $B>0$ there exists a small enough $c>0$ such that the right hand side of the preceding display with $a=c(n/\log n)^{1/(1+2\beta)}/\log n$ is bigger than $B$. It remained to deal with the first part of (F.2). We show below that with probability tending to one $\inf_{a\in[\underline{a}_{n}^{\prime}/2,\underline{a}_{n}^{\prime}]}\mathbb{M}_{n}(a)\geq cB\log^{2}(n/a)$, for some small enough constant $c>0$, not depending on $B$. Then with probability tending to one for any $a\in[1,\underline{a}_{n}^{\prime}/2]$ we have

[TABLE]

for large enough choice of $B>0$, hence the global maximum of $\ell_{n}(a)$ lies outside of the interval $[1,\underline{a}_{n}^{\prime}]$.

It remained to verify the lower bound for $M_{n}(a)$. First note that for $a\leq A_{n}=o(n)$

[TABLE]

hence $\underline{a}_{n}^{\prime}\leq c_{M,\beta}^{\prime}B^{-1/(1+2\beta)}(n/\log n)^{1/(1+2\beta)}/\log n$. Therefore in view of (F.3) for every $a\geq\underline{a}_{n}^{\prime}/2$ we have $g_{n}(a,f_{0})\geq c_{M,\beta,m}B$, for some positive constant $c_{M,\beta,m}>0$ not depending on $B$. Similarly we can show that $g_{n}(a,f_{0})\leq c_{M,\beta,m}^{\prime}B$, for every $a\geq\underline{a}_{n}^{\prime}/2$, for some $c_{M,\beta,m}^{\prime}>0$ not depending on $B$. Then following the same line of reasoning as in Section E.1, with the only main difference that instead of the interval given in (E.3) we are working with the interval $[\underline{a}_{n}^{\prime}/2,\underline{a}_{n}^{\prime}]$ we get that with porbability going to one

[TABLE]

for large enough choice of $M>0$, finishing the proof of the theorem.

Appendix G Proofs for the Hierarchical Bayes procedure

In this section we prove the results on the hierarchical Bayes procedure (i.e. Theorems A.5 and 2.4 and Corollary 2.1) based on the results derived for the empirical Bayes procedure. First we state that under the conditions of Theorem A.5 the hyper-posterior distribution on the hyper-parameter $a$ concentrates most of its mass on the interval $\mathcal{I}_{n}=[\underline{a}_{n}\log(n)/(1+\log n),C\overline{a}_{n}]$, for some large enough constant $C>0$.

Lemma G.1**.**

If $a\sim\pi(.)$ such that $\pi$ verifies Assumption 1 then for sufficiently large $C>0$ we have for every $\beta_{0}>0$ that

[TABLE]

G.1. Proof of Theorem A.5

Take $\varepsilon_{n}=(n/\log^{2}n)^{-\beta/(1+2\beta)}$. Then following from Lemma G.1, we have

[TABLE]

where the last equation follows by similar arguments as given in (A.9) and the displays below it (the only difference is that the supremum is taken over the interval $\mathcal{I}_{n}$ instead of $[\underline{a}_{n},\overline{a}_{n}]$, but it only changes the constant factors which do not play an essential role. This concludes the proof of the theorem.

G.2. Proof of Theorem 2.4 - Hierarchical Bayes part

Let us introduce the notations $W=\hat{f}-E_{0}\hat{f}$ and $B(f_{0})=E_{0}\hat{f}-f_{0}$, for the centered hierarchical posterior mean and the bias of the posterior mean, respectively. Then $P_{0}(f_{0}\in\hat{C}(L\log n))$ if and only if

[TABLE]

holds. Using assertions (B.2), (B.3), and (B.4) we show below that, there exist constants $\tilde{C}_{1},\tilde{C}_{2},\tilde{C}_{3}>0$, such that

[TABLE]

resulting in (G.1) for sufficiently large choice of $L>0$.

Proof of (G.2): Let us take any $\alpha^{\prime}>\alpha$ and note that in view of (B.2) we have

[TABLE]

Next, in view of Lemma G.1 and Anderson’s lemma, we get for arbitrary $r\leq\inf_{a\in\mathcal{I}_{n}}r_{\alpha^{\prime}}(a)$ that

[TABLE]

hence $r_{\alpha}^{2}\geq\inf_{a\in\mathcal{I}_{n}}r_{\alpha^{\prime}}(a)^{2}\geq C_{1}(\underline{a}_{n}/n)\log(n/\underline{a}_{n})$.

Proof of (G.3): Note that by triangle inequality, Fubini’s theorem, assertion (B.3), and Lemma G.1 we get that under the polished tail condition with $P_{0}$-probability tending to one

[TABLE]

where $\pi(\mathcal{I}_{n}|Y)$ denotes (by slightly abusing our notation) the posterior probability that the hyper-parameter $a$ lies in the interval $\mathcal{I}_{n}$ and in the last inequality we used in view of the proof of assertion (B.3) that $\sup_{1\leq a\leq A_{n}}\|W(a)\|_{2}=O(1)$.

Proof of (G.4): Similarly to the proof of (G.3) we get that

[TABLE]

where the last inequality follows from $\|B(a,f_{0})\|_{2}^{2}\leq\|f_{0}\|_{2}^{2}=O(1)$, finishing the proof of the theorem.

G.3. Proof of Corollary 2.1

Let $\varepsilon_{n}=(n/\log^{2}n)^{-\beta/(1+2\beta)}$ and first note that in view of assertions (G.3) and (G.4) combined with triangle inequality and Proposition A.2 we have with $P_{0}$-probability tending to one that

[TABLE]

Then in view of Theorem A.5 and by applying again the triangle inequality we get with probability tending to one that

[TABLE]

concluding the proof of the corollary.

G.4. Proof of Lemma G.1

In Section E it was shown that $\mathbb{M}_{n}(a)=\frac{\partial\ell_{n}(a)}{\partial a}$ satisfies, for positive constants $K_{1}$, $K_{2}$ and $K_{3}$,

[TABLE]

where $\underline{a}_{n}^{*}=\underline{a}_{n}\log n/(1+\log n)$. Furthermore, the constant $K_{2}$ can be chosen arbitrarily large by choosing $B$ large enough, while the constant $K_{3}$ is fixed.

For $a\geq C\overline{a}_{n}$ with $C\geq 3$, we have

[TABLE]

with $K_{4}=K_{1}(C-2)$. Consequently $e^{\ell_{n}(a)}\leq e^{\ell_{n}(2\overline{a}_{n})-K_{4}\log^{2}(n/\overline{a}_{n})\overline{a}_{n}}$ for $a\geq C\overline{a}_{n}$. Since also $e^{\ell_{n}(a)}\geq e^{\ell_{n}(2\overline{a}_{n})}$ for $a\in[\overline{a}_{n},2\overline{a}_{n}]$, we find

[TABLE]

Note that by Assumption 1

[TABLE]

hence the right hand side of (G.5) tends to zero.

The analysis of the left tail goes similarly. Note that for $a<\underline{a}_{n}^{*}/2$ we have $\ell_{n}(\underline{a}_{n}^{*})-\ell_{n}(a)\geq-K_{3}(\underline{a}_{n}^{*}-a)\log^{2}(n/\underline{a}_{n}),$ hence $e^{\ell_{n}(a)}\leq e^{\ell_{n}(\underline{a}_{n}^{*})+K_{3}\underline{a}_{n}\log^{2}(n/\overline{a}_{n})}$ and analogously for $(\underline{a}_{n}+\underline{a}_{n}^{*})/2<a<\underline{a}_{n}$ we have $\ell_{n}(a)-\ell_{n}(\underline{a}_{n}^{*})\geq K_{2}(a-\underline{a}_{n}^{*})\log^{3}(n/\underline{a}_{n})$, which implies $e^{\ell_{n}(a)}\geq e^{\ell_{n}(\underline{a}_{n}^{*})+K_{2}(\underline{a}_{n}/4)\log^{2}(n/\underline{a}_{n})}$. Therefore

[TABLE]

Since

[TABLE]

for large enough choice of $K_{2}$, the right hand side of (G.6) tends to zero, finishing the proof of the lemma.

Appendix H Technical Lemmas

Lemma H.1**.**

Let $i,m\in\mathbb{N}$ and $a\geq 1$, then for any $n/a\geq e^{m}$

[TABLE]

Proof.

Assume first that $i\leq I_{a}\equiv a\log(n/a)$. Note that the function $f(x)=e^{x/a}(x/a)^{m}$ is monotone decreasing on $(-\infty,-ma]$ and monotone increasing on $[-ma,\infty]$. Then by the inequality $ae^{i/a}+n\geq n$,

[TABLE]

Next assume that $i>I_{a}$. Note that the derivative of the function $f(x)=e^{-x/a}x^{m}$ is $f^{\prime}(x)=e^{-x/a}x^{m-1}(m-x/a)$, hence the function $f(i)$ is monotone decreasing for $i\geq am$. Thus for $n/a\geq e^{m}$, $f(i)$ takes its maximum at $i=I_{a}$, which implies that

[TABLE]

∎

Lemma H.2**.**

Let $l>r\geq 0$, then for $n/a\geq e^{l-r}$

[TABLE]

Proof.

First note that following from the inequality $ae^{i/a}+n\geq n$ and the sum of geometric series we get

[TABLE]

where $I_{a}\equiv a\log(n/a)$. Then similarly, using the inequality $ae^{i/a}+n\geq ae^{i/a}$ and the sum of geometric series,

[TABLE]

because $e^{(l-r)/a}-1\geq\frac{l-r}{a}$ and \log\big{(}\frac{n}{a}\big{)}\geq l-r for $\frac{n}{a}\geq e^{l-r}$. ∎

Lemma H.3** (Lemma C.11 of [30]).**

For any stochastic process $(V_{a}:a>0)$ with continuously differentiable sample paths $a\mapsto V_{a}$, with derivative written as $\dot{V}_{a}$,

[TABLE]