On One Property of Tikhonov Regularization Algorithm

TL;DR

This paper demonstrates that Tikhonov regularization is minimax optimal for linear inverse problems with Gaussian noise, especially when the solution's Fourier coefficients satisfy certain a priori conditions, including membership in a Besov space.

Contribution

It establishes the minimax optimality and asymptotic minimaxity of Tikhonov regularization under specific a priori Fourier coefficient conditions.

Findings

Tikhonov regularization is minimax in the class of linear estimators.

It is asymptotically minimax among all estimators.

Results depend on a priori information about Fourier coefficients, especially in Besov spaces.

Abstract

For linear inverse problem with Gaussian random noise we show that Tikhonov regularization algorithm is minimax in the class of linear estimators and is asymptotically minimax in the sense of sharp asymptotic in the class of all estimators. The results are valid if some a priori information on a Fourier coefficients of solution is provided. For trigonometric basis this a priori information implies that the solution belongs to a ball in Besov space $B^r_{2\infty}$.

Equations

$$f=Ax,x\in H,$$

$$y_i=a_i{x}_i+ϵ{\sigma }_i{\xi }_i$$

$$x\in B_{2\infty }^r=\left\{\theta =\{{\theta }_i{\}}_{i=1}^{\infty }:\underset{k}{\sup }{k}^{2r}\sum _{i=1}^{\infty }{\theta }_i^2\le P_0\right\}$$

$$\underset{x\in B_{2\infty }^r}{\sup }{E}_x||{\hat{x}}_{ϵ}-x|{|}^2=\underset{\lambda }{\inf }\underset{x\in B_{2\infty }^r}{\sup }{E}_x||{\hat{x}}_{ϵ\lambda }-x|{|}^2.$$

$$\underset{x\in B_{2\infty }^r}{\sup }{E}_x||{\hat{x}}_{ϵ}-x|{|}^2=\underset{{\tilde{x}}_{ϵ}\in \Psi }{\inf }\underset{x\in B_{2\infty }^r}{\sup }{E}_x||{\tilde{x}}_{ϵ}-x|{|}^2(1+o(1))$$

$$\frac{{\sigma }_j^2{a}_{j-1}^2((j-1{)}^{-2r}-j^{-2r})}{{\sigma }_{j-1}^2{a}_j^2(j^{-2r}-(j+1{)}^{-2r})}>1.$$

$${\lambda }_j=a_j^{-1}\frac{a_j^2{P}_0(j^{-2r}-(j+1{)}^{-2r})}{a_j^2{P}_0(j^{-2r}-(j+1{)}^{-2r})+{ϵ}^2{\sigma }_j^2}.$$

$$R_{lϵ}={ϵ}^2(1+o(1))\sum _{j=1}^{\infty }\frac{{\sigma }_j^2{P}_0(j^{-2r}-(j-1{)}^{-2r})}{a_j^2{P}_0(j^{-2r}-(j-1{)}^{-2r})+{ϵ}^2{\sigma }_j^2}.$$

$$\frac{{\sigma }_j^2{a}_{j-1}^2{j}^{2r+1}}{{\sigma }_{j-1}^2{a}_j^2(j-1{)}^{2r+1}}>1.$$

$${\lambda }_j=a_j^{-1}\frac{a_j^2}{a_j^2+(2r{P}_0{)}^{-1}{ϵ}^2{\sigma }_j^2{j}^{2r+1}}.$$

$$R_{ϵ}={ϵ}^2(1+o(1))\sum _{j=1}^{\infty }\frac{{\sigma }_j^2}{a_j^2+(2r{P}_0{)}^{-1}{ϵ}^2{\sigma }_j^2{j}^{2r+1}}.$$

$$R_{ϵ}={ϵ}^{\frac{4r}{1+2r+2\gamma }}\frac{\pi }{2r\sin \left(\frac{\pi (2\gamma +1)}{2r}\right)}(2r{P}_0{)}^{\frac{2\gamma +1}{2\gamma +2r+1}}{C}^{-\frac{2r}{2\gamma +2r+1}}(1+o(1)).$$

$$R_{ϵ}=|\log ϵ{|}^{-2r/\gamma }{P}_0{B}^{2r/\gamma }(1+o(1)).$$

$$\begin{gathered} \underset{\lambda }{\inf }\underset{x\in B_{2\infty }^r}{\sup }{E}_x||{\hat{x}}_{\lambda }-x|{|}^2\ge \underset{\lambda }{\inf }{E}_{\theta }||{\hat{\theta }}_{\lambda }-\theta |{|}^2 \\ ={ϵ}^2\sum _{j=1}^{\infty }\frac{{\theta }_j^2{\sigma }_j^2}{{\theta }_j^2{a}_j^2+{ϵ}^2{\sigma }_j^2} \end{gathered}$$

$${\lambda }_j=a_j^{-1}\frac{a_j^2{\theta }_j^2}{{\theta }_j^2{a}_j^2+{ϵ}^2{\sigma }_j^2}.$$

$$u_k=k^{2r}\sum _{j=k}^{\infty }{x}_j^2.$$

$$\begin{array}{rl}& E_x\sum _{j=1}^{\infty }({\lambda }_j{y}_j-x_j{)}^2={ϵ}^2\sum _{j=1}^{\infty }{\lambda }_j^2{\sigma }_j^2{a}_j^{-2}+\sum _{j=1}^{\infty }(1-a_j{\lambda }_j{)}^2{x}_j^2\\ & ={ϵ}^2\sum _{j=1}^{\infty }{\lambda }_j^2{\sigma }_j^2{a}_j^{-2}+\sum _{j=1}^{\infty }{\left(\frac{1}{{\theta }_j^2{\sigma }_j^{-2}{a}_j^2{ϵ}^{-2}+1}\right)}^2(j^{-2r}{u}_j-(j+1{)}^{-2r}{u}_{j+1})\\ & ={ϵ}^2\sum _{j=1}^{\infty }{\lambda }_j^2{\sigma }_j^2{a}_j^{-2}+{\left(\frac{1}{{\theta }_1^2{\sigma }_1^{-2}{a}_1^2{ϵ}^{-2}+1}\right)}^2{u}_1\\ & -\sum _{j=2}^{\infty }{u}_j{j}^{-2r}\left(({\theta }_{j-1}^2{\sigma }_{j-1}^{-2}{a}_{j-1}^2{ϵ}^{-2}+1{)}^{-2}-({\theta }_j^2{\sigma }_j^{-2}{a}_j^2{ϵ}^{-2}+1{)}^{-2}\right).\end{array}$$

$$\begin{array}{rl}& \underset{x\in B_{2\infty }^s}{\sup }{E}_x||\hat{x}-x|{|}^2\ge E_{{\nu }_{\delta }}{E}_{\theta }||\hat{x}-\theta |{|}^2\\ & \ge E_{\mu }{E}_{\eta }||\tilde{x}-\eta |{|}^2-E_{\mu }{E}_{\eta }(||\bar{x}-\eta |{|}^2,\eta \notin B_{2\infty }^r)P_{\mu }^{-1}(\eta \in B_{2\infty }^r).\end{array}$$

$$E_{\mu }{E}_{\eta }||\tilde{x}-\eta |{|}^2={ϵ}^2(1+o(1))\sum _{j=l_1}^{l_2}\frac{{\sigma }_j^2}{a_j^2+(2r(P_0-{\delta }_1){)}^{-1}{ϵ}^2{\sigma }_j^2{j}^{2r+1}}\doteq I(P_0-{\delta }_1)$$

$$||\bar{x}|{|}^2\le \underset{x\in B_{2\infty }^r}{\sup }||x|{|}^2\le P_0,$$

$$\begin{array}{rl}& E_{\mu }{E}_{\eta }(||\bar{x}-\eta |{|}^2,\eta \notin B_{2\infty }^r)\le 2E_{\mu }{E}_{\eta }(||\bar{x}|{|}^2+||\eta |{|}^2,\eta \notin B_{2\infty }^r)\\ & \le 2P_0{P}_{\mu }(\eta \notin B_{2\infty }^r)+\sum _{j=l_1}^{l_2}(E_{\mu }{\eta }_j^4{)}^{1/2}{P}_{\mu }^{1/2}(\eta \notin B_{2\infty }^r).\end{array}$$

$$\sum _{j=l_1}^{l_2}(E_{\mu }{\eta }_j^4{)}^{1/2}\le C{\delta }^{-r}{k}_{ϵ}^{-2r}.$$

$$P_{\mu }(\eta \notin B_{2\infty }^r)=P(\underset{l_1\le i\le l_2}{\max }{i}^{2r}\sum _{j=i}^{l_2}{\eta }_j^2-P_0(1-{\delta }_1/2)>P_0{\delta }_1/2)\le \sum _{i=l_1}^{l_2}{J}_i$$

$$J_i=P\left(i^{2r}\sum _{j=i}^{l_2}{\eta }_j^2-P_0(1-{\delta }_1/2)>P_0{\delta }_1/2\right)$$

$$P(||A\xi |{|}^2>\text{mbox}tr(\Sigma )+2\sqrt{\text{mbox}tr({\Sigma }^2)t}+2||\Sigma ||t)\le \exp \{-t\}.$$

$$2\sqrt{\text{mbox}tr({\Sigma }^2)t}+2||\Sigma ||t=\frac{P_0-\delta }{r(4r+1)}\sqrt{i^{-1}t}(1+o(1))+i^{-1}t\doteq V_i(t)$$

$$J_i<\exp \{-k_{ϵ}^{-1/2}\}$$

$$\sum _{j=l_1}^{l_2}{J}_i\le {\delta }^{-1}{k}_{ϵ}\exp \{-k_{ϵ}^{1/2}\}$$

$$|I(P_0)-I(P_0-{\delta }_1)|<C{\delta }_{1}I(P_0)$$

$$\begin{array}{rl}& {ϵ}^2\sum _{j=1}^{l_1}\frac{{\sigma }_j^2}{a_j^2+(2r{P}_0{)}^{-1}{ϵ}^2{\sigma }_j^2{j}^{2r+1}}\asymp {ϵ}^2\sum _{j=1}^{l_1}{\sigma }_j^2{a}_j^{-2}\\ & <C{\delta }_1\sum _{j=l_1}^{k_{ϵ}}{\sigma }_j^2{a}_j^{-2}\asymp C{\delta }_1{ϵ}^2\sum _{j=l_1}^{k_{ϵ}}\frac{{\sigma }_j^2}{a_j^2+(2r{P}_0{)}^{-1}{ϵ}^2{\sigma }_j^2{j}^{2r+1}}\end{array}$$

Full text

erm2512@gmail.com

Institute of Problems of Mechanical Engineering, RAS and

St. Petersburg State University, St. Petersburg, RUSSIA

Mechanical Engineering Problems Institute

Russian Academy of Sciences

Bolshoy pr.,V.O., 61

St.Petersburg

Russia St.Petersburg State University

University pr., 28, Petrodvoretz

198504 St.Petersburg

Russia St.Petersburg department of

Steklov mathematical institute

Fontanka 27, St.Petersburg 191023

1. **Introduction **

Tikhonov regularization algorithm (TRA) is very popular [16, 13] thanks to many remarkable properties. We mention only two of them. TRA is minimax for deterministic noise [12] and is Bayes estimator [17, 3] in the problems with Gaussian random noise and Gaussian a priori measure. In paper we explore the minimax properties of TRA in linear inverse problems with Gaussian random noise. We show that TRA is minimax in the class of linear estimator and asymptotically minimax in the class of all estimators. In these setups a priori information is provided that Fourier coefficients of solution satisfies the same restrictions as Fourier coefficients of functions in a ball in Besov space $B^{s}_{2\infty}$ for the case of trigonometric basis.

Such a form of a priori information is rather natural. This is rather reasonable information on a solution smoothness. For the most nonparametric estimators these sets are the largest sets with a given rates of convergence [8, 9]. For linear statistical estimators these sets are the largest sets with a given rate of convergence [15]. The asymptotic minimaxity of TRA is proved in the sense of sharp asymptotic. The asymptotically minimax nonparametric estimators in the sense of sharp asymptotic has been obtained earlier, only if a priori information is provided that a solution belongs to ellipsoid in $L_{2}$, in particular, a ball in Sobolev space [14, 18, 7].

There are numerous research on sharp adaptive minimax estimation [3, 7, 2, 18, 11]. The results on adaptive estimation in Pinsker model [7, 18] are easily carried over on paper setup.

In what follows we shall denote letters $c,C$ positive constants and $a_{\epsilon}\asymp b_{\epsilon}$ implies $c<a_{\epsilon}/b_{\epsilon}<C$.

2. Main Results

Let $H$ be separable Hilbert space and let $A:H\to H$ be known self-adjoint linear bounded operator.

We wish to estimate a solution of linear equation

[TABLE]

on observation $Y=f+\epsilon\xi$ where $\xi$ is Gaussian random error and $\epsilon>0$ defines the level of noise.

Let $\{a_{i}\}_{i=1}^{\infty}$ and $\{\phi_{i}\}_{i=1}^{\infty}$ be eigenvalues and eigenvectors of operator $A$ respectively. Then we can rewrite the vector $Y=\{y_{i}\}_{i=1}^{\infty}$ in the following form

[TABLE]

where $x_{i}=<x,\phi_{i}>$, $y_{i}=<Y,\phi_{i}>$ and $\xi_{i}=\sigma_{i}^{-1}<\xi,\phi_{i}>,1\leq i<\infty$, $Var[\xi_{i}]=1$. We suppose that $\xi_{i},1\leq i<\infty$ are i.i.d.r.v.’s and $E[\xi_{i}]=0$. The representation (2.1) holds in particular if $\xi$ is Gaussian white noise.

Here $<a,b>$ denote inner product of vectors $a,b\in H$. For any $a\in H$ denote $||a||=<a,a>^{1/2}$.

Suppose a priori information is provided that

[TABLE]

with $r>0$.

We say that linear estimator $\hat{x}_{\epsilon}=\{\hat{x}_{\epsilon j}\}_{j=1}^{\infty}$ is minimax linear estimator if

[TABLE]

where $\lambda=\{\lambda_{j}\}_{j=1}^{\infty}$, $\hat{x}_{\epsilon\lambda}=\{\hat{x}_{\epsilon j\lambda_{j}}\}_{j=1}^{\infty},\hat{x}_{\epsilon j\lambda_{j}}=\lambda_{j}y_{j},\lambda_{j}\in R^{1},1\leq j<\infty.$

We say that the estimator $\hat{x}_{\epsilon}$ is asymptotically minimax if

[TABLE]

as $\epsilon\to 0$. Here $\Psi$ is the set of all estimators.

The minimaxity of TRA in the class of linear estimators will be proved if the following assumption holds.

A. For all $j>1$

[TABLE]

Theorem 2.1**.**

Assume A. Then TRA is minimax on the set of all linear estimators with

[TABLE]

The asymptotically minimax risk equals

[TABLE]

The asymptotic minimaxity of TRA will be proved if the following assumptions hold.

B1 For $j>j_{0}$, there holds $|a_{j}/a_{j+1}|\geq 1$.

B2 There holds $0<c<\sigma_{j}^{2}<C<\infty$.

B3. For all $j>j_{0}$

[TABLE]

Theorem 2.2**.**

Assume B1-B3. Then TRA is asymptotically minimax on the set of all estimators with

[TABLE]

The asymptotically minimax risk equals

[TABLE]

Example 2.1**.**

Let $|a_{j}|=Cj^{-\gamma}(1+o(1))$ and $\sigma_{j}=1$. Then

[TABLE]

Example 2.2**.**

Let $|a_{j}|=Cj^{-\alpha}\exp\{-Bj^{\gamma}\}$ and $\sigma_{j}=1$. Then

[TABLE]

3. Proof of Theorem 2.1

We begin with the proof of lower bound. Denote $\theta_{j}^{2}=P_{0}(j^{-2r}-(j+1)^{-2r}),\theta=\{\theta_{j}\}_{j=1}^{\infty}$.

We have

[TABLE]

and infimum is attained for

[TABLE]

Proof of upper bound is based on the following reasoning. Let $x=\{x_{j}\}_{j=1}^{\infty}\in B^{r}_{2\infty}$. For all $k$ denote

[TABLE]

Then $x_{k}^{2}=k^{-2r}u_{k}-(k+1)^{-2r}u_{k+1}$.

For the sequence of $\lambda_{j}$ defined in Theorem 2.1, we have

[TABLE]

By A, the last addendums in the right hand-side of (3.2) are negative. Therefore the supremum of right hand-side of (3.2) is attained for $u_{j}=P_{0}$, $1\leq j<\infty$. This completes the proof of Theorem (2.1).

4. Proof of Theorem 2.2

The upper bound follows from Theorem 2.1. Below the proof of lower bound will be provided. This proof has a lot of common feachers with the proof of lower bound in Pinsker Theorem [7, 14, 18].

Fix values $\delta,0<\delta<1,$ and $\delta_{1},0<\delta_{1}<P_{0}$. Define a family of natural numbers $k_{\epsilon},\epsilon>0,$ such that $a_{k_{\epsilon}}^{2}\epsilon^{-2}\sigma^{2}_{k_{\epsilon}}P_{0}(2r)^{-1}k_{\epsilon}^{-2r-1}=1+o(1)$ as $\epsilon\to 0$. Define sequence $\eta=\{\eta_{j}\}_{j=1}^{\infty}$ of Gaussian i.i.d.r.v.’s $\eta_{j}=\eta_{j\delta\delta_{1}},E[\eta_{j}]=0,\mbox{Var}[\eta_{j}]=(P_{0}-\delta_{1})(2r)^{-1}j^{-2r-1}$, if $\delta k_{\epsilon}\leq j\leq\delta^{-1}k_{\epsilon}$, and $\eta_{j}=0$ for $j<\delta k_{\epsilon}$ and $j>\delta^{-1}k_{\epsilon}$

Denote $\mu$ the probability measure of random vector $\eta$. Define $\tilde{x}$ Bayes estimator with a priory measure $\mu$.

Define the conditional probability measure $\nu_{\delta}$ of random vector $\eta$ given $\eta\in B^{r}_{2\infty}(P_{0}).$ Define $\bar{x}$ Bayes estimator $x$ with a priori measure $\nu_{\delta}$. Denote $\theta$ the random variable having probability measure $\nu_{\delta}$.

For any estimator $\hat{x}$ we have

[TABLE]

We have

[TABLE]

with $l_{1}=[\delta k_{\epsilon}]$ and $l_{2}=[\delta^{-1}k_{\epsilon}].$ Here $[a]$ denotes whole part of a number $a\in R^{1}$.

Since

[TABLE]

we have

[TABLE]

Since $E_{\mu}[\eta_{j}^{4}]\leq Cj^{-2r-2},$ we have

[TABLE]

It remains to estimate

[TABLE]

with

[TABLE]

To estimate $J_{i}$ we implement the following Proposition [6]

Proposition 4.1**.**

Let $\xi=\{\xi_{i}\}_{i=1}^{l}$ be Gaussian random vector with i.i.d.r.v.’s $\xi_{i}$, $E[\xi_{i}]=0,E[\xi_{i}^{2}]=1$. Let $A\in R^{l}\times R^{l}$ and $\Sigma=A^{T}A$. Then

[TABLE]

We put $\Sigma=\{\sigma_{lj}\}_{l,j=i}^{l_{2}}$ with $\sigma_{jj}=j^{-2r-1}i^{2r}\frac{P_{0}-\delta}{2r}$ and $\sigma_{lj}=0$ if $l\neq j$. Then

[TABLE]

Since $i>\delta k_{\epsilon}$, we can put $t=k_{\epsilon}^{1/2}$. Then $V_{i}(t)<Ck_{\epsilon}^{-1/2}$ and implementing (4.6) we have

[TABLE]

and therefore

[TABLE]

To complete the proof it remains to estimate $R_{\epsilon}-I(P_{0}-\delta_{1})$.

By straightforward estimation, it is easy to verify that

[TABLE]

We have

[TABLE]

We have

[TABLE]