Convergence of Heuristic Parameter Choice Rules for Convex Tikhonov Regularisation

TL;DR

This paper extends the convergence analysis of heuristic parameter choice rules for convex Tikhonov regularisation, demonstrating their effectiveness under noise restrictions and providing theoretical and numerical insights.

Contribution

It introduces new convergence results for heuristic rules in convex Tikhonov regularisation, expanding the linear theory to the convex setting with practical examples.

Findings

Convergence of heuristic rules under noise restrictions is established.

Numerical examples illustrate the theoretical results.

Analysis applies to ill-posed problems with diagonal operators in ^q spaces.

Abstract

We investigate the convergence theory of several known as well as new heuristic parameter choice rules for convex Tikhonov regularisation. The success of such methods is dependent on whether certain restrictions on the noise are satisfied. In the linear theory, such conditions are well understood and hold for typically irregular noise. In this paper, we extend the convergence analysis of heuristic rules using noise restrictions to the convex setting and prove convergence of the aforementioned methods therewith. The convergence theory is exemplified for the case of an ill-posed problem with a diagonal forward operator in $\ell^q$ spaces. Numerical examples also provide further insight.

Equations

$$Ax=y,$$

$$x_{\alpha }^{\delta }\in \underset{x\in X}{\mathrm{argmin}}{\mathcal{T}}_{\alpha }^{\delta }(x),$$

$${\mathcal{T}}_{\alpha }^{\delta }(x):=\frac{1}{2}\Vert Ax-y^{\delta }{\Vert }^2+\alpha \mathcal{R}(x),$$

$$\alpha ={\alpha }_{\ast }\in \underset{\alpha \in (0,{\alpha }_{\text{max}})}{\mathrm{argmin}}\psi (\alpha ,y^{\delta }),$$

$$\psi :(0,{\alpha }_{\text{max}})\times Y\to \mathbb{R}.$$

$${\psi }_{\text{HD}}(\alpha ,y^{\delta })$$

$${\psi }_{\text{HR}}(\alpha ,y^{\delta })$$

$${\psi }_{\text{SQO}}(\alpha ,y^{\delta })$$

$${\psi }_{\text{RQO}}(\alpha ,y^{\delta })$$

$$D_{\xi }(x_{\alpha }^{\delta },x^{†}):=\mathcal{R}(x_{\alpha }^{\delta })-\mathcal{R}(x^{†})-⟨\xi ,x_{\alpha }^{\delta }-x^{†}{⟩}_{X^{\ast }\times X}.$$

$$D_{\xi }(x_{\alpha }^{\delta },x^{†})=D_{{\xi }_{\alpha }}(x_{\alpha }^{\delta },x_{\alpha })+D_{\xi }(x_{\alpha },x^{†})+⟨{\xi }_{\alpha }-\xi ,x_{\alpha }^{\delta }-x_{\alpha }⟩.$$

$$D_{{\xi }_1,{\xi }_2}^{\text{sym}}:X\times X$$

$$(x_1,x_2)$$

$$x_{\alpha }^{\delta }\in \underset{x\in X}{\mathrm{argmin}}{\mathcal{T}}_{\alpha }^{\delta }(x)\text{and}{x}_{\alpha }\in \underset{x\in X}{\mathrm{argmin}}{\mathcal{T}}_{\alpha }(x),$$

$$p_{\alpha }^{\delta }:=y^{\delta }-A{x}_{\alpha }^{\delta }\text{and}{p}_{\alpha }:=y-A{x}_{\alpha }.$$

$${\mathrm{prox}}_{\mathcal{J}}:Y\to Y,{\mathrm{prox}}_{\mathcal{J}}=(I+\partial \mathcal{J}{)}^{-1},$$

$$p_{\alpha }^{\delta }:={\mathrm{prox}}_{\mathcal{J}}(y^{\delta }),\text{with }\mathcal{J}=\alpha {\mathcal{R}}^{\ast }\circ \frac{1}{\alpha }{A}^{\ast }.$$

$$⟨{\mathrm{prox}}_{\mathcal{J}}(y_1)-{\mathrm{prox}}_{\mathcal{J}}(y_2),y_1-y_2⟩\ge \Vert {\mathrm{prox}}_{\mathcal{J}}(y_1)-{\mathrm{prox}}_{\mathcal{J}}(y_2){\Vert }^2,$$

$$0\in A^{\ast }(A{x}_{\alpha }^{\delta }-y^{\delta })+\alpha \partial \mathcal{R}(x_{\alpha }^{\delta }),$$

$${\xi }_{\alpha }:=\partial \mathcal{R}(x_{\alpha }^{\delta })=-\frac{1}{\alpha }{A}^{\ast }(A{x}_{\alpha }-y)\text{and}{\xi }_{\alpha }^{\delta }:=\partial \mathcal{R}(x_{\alpha })=-\frac{1}{\alpha }{A}^{\ast }(A{x}_{\alpha }^{\delta }-y^{\delta }),$$

$$x_{\alpha ,\delta }^{II}\in \underset{x\in X}{\mathrm{argmin}}\frac{1}{2}\Vert Ax-y^{\delta }{\Vert }^2+\alpha D_{{\xi }_{\alpha }^{\delta }}(x,x_{\alpha }^{\delta }).$$

$$0\in A^{\ast }(A{x}_{\alpha ,\delta }^{II}-y^{\delta }-(y^{\delta }-A{x}_{\alpha }^{\delta }))+\alpha \partial \mathcal{R}(x_{\alpha ,\delta }^{II}),$$

$${\xi }_{\alpha }^{II}:=-\frac{1}{\alpha }{A}^{\ast }(A{x}_{\alpha }^{II}-y-(y-A{x}_{\alpha }))\text{and}{\xi }_{\alpha ,\delta }^{II}:=-\frac{1}{\alpha }{A}^{\ast }(A{x}_{\alpha ,\delta }^{II}-y^{\delta }-(y^{\delta }-A{x}_{\alpha }^{\delta })),$$

$$p_{\alpha ,\delta }^{II}:=y^{\delta }-A{x}_{\alpha ,\delta }^{II}={\mathrm{prox}}_{\mathcal{J}}\left(y^{\delta }+p_{\alpha }^{\delta }\right)-p_{\alpha }^{\delta }\text{and}{p}_{\alpha }^{II}:=y-A{x}_{\alpha }^{II}={\mathrm{prox}}_{\mathcal{J}}\left(y+p_{\alpha }\right)-p_{\alpha },$$

$$\Delta y=y^{\delta }-y,\Delta p_{\alpha }=p_{\alpha }^{\delta }-p_{\alpha },\Delta p_{\alpha }^{II}=p_{\alpha ,\delta }^{II}-p_{\alpha }^{II}.$$

$$D_{{\xi }_{\alpha }}(x_{\alpha }^{\delta },x_{\alpha })\le \frac{1}{\alpha }⟨\Delta y-\Delta p_{\alpha },\Delta p_{\alpha }⟩$$

$$D_{{\xi }_{\alpha }}(x_{\alpha }^{\delta },x_{\alpha })$$

$$\le -\frac{1}{\alpha }⟨A(x_{\alpha }^{\delta }-x_{\alpha }),A(x_{\alpha }^{\delta }-x_{\alpha })⟩-\frac{1}{\alpha }⟨y-y^{\delta },A(x_{\alpha }^{\delta }-x_{\alpha })⟩$$

$$=-\frac{1}{\alpha }⟨A{x}_{\alpha }^{\delta }-y^{\delta }+y-A{x}_{\alpha },A(x_{\alpha }^{\delta }-x_{\alpha })⟩,$$

$$\partial \mathcal{R}(x^{†})\in \mathrm{range}(A^{\ast })⟺\exists w:A^{\ast }w\in \partial \mathcal{R}(x^{†}).$$

$$D_{\xi }(x_{\alpha },x^{†})$$

$$D_{{\xi }_{\alpha }}(x_{\alpha }^{\delta },x_{\alpha })$$

$$D_{\xi }(x_{\alpha }^{\delta },x^{†})\le \frac{1}{2}{\left(\frac{\delta }{\sqrt{\alpha }}+\sqrt{\alpha }\Vert w\Vert \right)}^2,\Vert A{x}_{\alpha }^{\delta }-y^{\delta }\Vert \le \delta +2\Vert w\Vert \alpha ,$$

$$\Vert A{x}_{\alpha ,\delta }^{II}-y^{\delta }\Vert \le \Vert A{x}_{\alpha }^{\delta }-y^{\delta }\Vert \text{and}\mathcal{R}(x_{\alpha }^{\delta })\le \mathcal{R}(x_{\alpha ,\delta }^{II}),$$

Full text

Keywords: ill-posed problems, convex regularisation, heuristic parameter choice rules

1 Introduction

Let $X$ and $Y$ be Banach and Hilbert spaces, respectively. We consider the ill-posed problem

[TABLE]

where $A:X\to Y$ is a continuous linear operator and only noisy data $y^{\delta}=y+e$ is available and $\delta$ such that $\|e\|\leq\delta$ is defined to be the noise level. In other words, we assume that the data does not depend continuously on the solution. We therefore determine a regularised solution à la Tikhonov:

[TABLE]

where

[TABLE]

is the Tikhonov functional, and the regularisation term given by the functional $\mathcal{R}:X\to\mathbb{R}\cup\{\infty\}$ is assumed to be convex, proper, coercive, weak-$\ast$ lower semicontinuous and $\alpha\in(0,\alpha_{\text{max}})$ is the so-called regularisation parameter. The aforementioned properties ensure the existence of a minimiser for $\mathcal{T}^{\delta}_{\alpha}$ (cf. [33]). In this way, we seek to approximate an $\mathcal{R}$-minimising solution, $x^{\dagger}\in X$ (cf. [33]).

The choice of regularisation parameter is pivotal for any reasonable approximation of $x^{\dagger}$. There are several classes of rules which select this parameter (cf., e.g., [13]), the best-known being the a-posteriori rules which select $\alpha_{\ast}:=\alpha(\delta,y^{\delta})$ in dependence of the noise level and the measured noisy data. A classic example is Morozov’s discrepancy principle (cf. [29]). The drawback is that in practical situations, the noise level is usually unknown. In this case, one may opt to use another class of parameter choice rules, namely the so-called heuristic rules, which select $\alpha_{\ast}:=\alpha(y^{\delta})$ in dependence of the measured data alone (i.e., without knowledge of the noise level). Their pitfall, however, comes in the form of the Bakushinskii veto (cf. [1]), which essentially asserts that a heuristic rule cannot yield a convergent regularisation method in the worst case scenario. However, it was proven recently in [25, 21] for linear regularisation methods that certain heuristic rules yield a convergent method if some noise conditions are postulated. It is important to note that the aforementioned noise conditions utilised the spectral theory for self-adjoint linear operators. A recent discussion and extension of the noise conditions within the linear theory may be found in [27].

The topic of this paper is the corresponding analysis for the convex case. Note that the tools from the linear theory are no longer applicable due to the absence of spectral theory. For the mentioned setting, some heuristic parameter choice rules were considered in the literature: B. Jin and Lorenz in [18] discussed the heuristic discrepancy rule and a version of the discrete quasi-optimality rule. The heuristic discrepancy rule was also considered for the augmented Lagrangian method and Bregman iteration for nonlinear operators in the work of Q. Jin (c.f. [19, 20, 35]). A numerical study of certain heuristic rules was investigated in [24].

Crucial to the convergence theory of heuristic rules are restrictions on the noise. In the linear theory, they take the form of Muckenhoupt-type conditions. In the convex case, some rather abstract conditions were proposed in [18, 19]. However, the validity of these conditions remains unclear.

In this paper, we propose several heuristic rules and, as main contribution, provide a convergence analysis by postulating so-called auto-regularisation conditions. They reduce to Muckenhoupt-type conditions in the setting of $\ell^{q}$ regularisation with a diagonal operator $A$, allowing us to subsequently investigate their validity for typical cases. The main results are Theorems (2), (3), (4) in Section (2) (with abstract conditions), and Theorems (6), (7), (8) (specific convergence conditions for the diagonal case) in Section (3). Furthermore, we provide a detailed numerical case study of these heuristic methods in Section (4).

2 Heuristic parameter choice rules

The heuristic rules we consider select the parameter $\alpha$ in the Tikhonov functional (1)

[TABLE]

as the global minimiser of a functional

[TABLE]

We investigate the following four functionals:

[TABLE]

where ${x_{\alpha,\delta}^{II}}$ is the second Bregman iterate, $D^{\text{sym}}$ and $D$ denote the symmetric and regular Bregman distances, all of which will be defined in the following.

Note that the heuristic discrepancy rule is sometimes also referred to as the Hanke-Raus rule (as the rules coincide for Landweber iteration). For clarity, it is preferable to name this method as the heuristic analogue of the classical discrepancy rule. In particular, this rule is the only one for which some convergence analysis has been done (cf. [18, 20, 19, 35]). A discrete version of the quasi-optimality rules was also investigated in [18]. As mentioned, a numerical study of some rules was also done in [24].

These rules are well established in the linear case. However, except for the HD rule, their extension to the convex setting is certainly not obvious. We consider Bregman iteration as the natural analogue of Tikhonov iteration and we opt to define the latter three rules utilising the second Bregman iterate. The HR rule was considered in [24], whilst the quasi-optimality rules considered here are entirely novel.

Note that beside the right quasi-optimality rule it is possible to define a “left”quasi-optimality version using the functional $D_{\xi^{II}_{\alpha,\delta}}(x^{\delta}_{\alpha},x^{II}_{\alpha,\delta})$. However, the numerical results for this rule were quite subpar (as was demonstrated in [24]), so we decided not to consider it further.

2.1 Preliminaries

Note that in contrast with linear regularisation theory, one cannot (in general) prove convergence of the regularised solution to $x^{\dagger}$ in the norm, but it is common to use the Bregman distance (cf. [6, 7]):

[TABLE]

The Bregman distance is not a distance (a.k.a. a metric) as it does not satisfy the triangle inequality; nor is it in general symmetric. We do, however, have the following useful so-called three point identity (cf. [23]):

[TABLE]

For any $\xi_{1}\in\partial\mathcal{R}(x_{1})$ and $\xi_{2}\in\partial\mathcal{R}(x_{2})$, one can also define the symmetric Bregman distance as

[TABLE]

In the following, we use a super/sub-scripted $\delta$ to indicate variables associated with noisy data and its absence indicates the corresponding variables for exact data. For instance,

[TABLE]

where $\mathcal{T}_{\alpha}$ indicates the Tikhonov functional with exact data $y$ replacing $y^{\delta}$. It is useful to define residuals as variables. In particular,

[TABLE]

The following proposition can be proven via standard convex analysis:

Proposition 1**.**

The residual ${p_{\alpha}^{\delta}}$ may be expressed in terms of a proximal mapping operator,

[TABLE]

in the form

[TABLE]

Note that in the above proposition, $A^{\ast}:Y\to X^{\ast}$ denotes the adjoint operator and $\mathcal{R}^{\ast}$ denotes the Fenchel conjugate (cf. [5, 4]). We will make use of the firm non-expansivity of the proximal mapping operator:

[TABLE]

cf. [5, 4].

The optimality condition for the Tikhonov functional may be stated as follows:

[TABLE]

for all $\alpha\in(0,\alpha_{\text{max}})$ and $y^{\delta}\in Y$. We may thus define

[TABLE]

for the subgradients of $\mathcal{R}$ at $x_{\alpha}$ and $x^{\delta}_{\alpha}$, respectively.

The nonlinear analogue of iterated Tikhonov regularisation, as stated earlier, may be defined as Bregman iteration (cf. [6, 30]). In particular, the second Bregman iterate may be computed as

[TABLE]

As observed in [34], we can also compute $x^{II}_{\alpha,\delta}$ by minimising a simpler expression which does not involve the Bregman distance. Similarly as for the Tikhonov functional, we can state the optimality condition for the subsequent Bregman functional in the same manner:

[TABLE]

and analogously, $x^{II}_{\alpha}$ for exact data. As before, we introduce the corresponding expressions for the subgradients

[TABLE]

of $\mathcal{R}$ at $x^{II}_{\alpha}$ and $x^{II}_{\alpha,\delta}$, respectively.

The residual with respect to the second Bregman iterate may also be written in terms of the proximal point mapping:

[TABLE]

for all $\alpha\in(0,\alpha_{\text{max}})$ and $y^{\delta}\in Y$ and $\mathcal{J}$ as in (4). For notational purposes, we define the following

[TABLE]

We state a useful estimate:

Lemma 1**.**

We have the following upper bound for the data propagation error:

[TABLE]

for all $\alpha\in(0,\alpha_{\text{max}})$ and $y,y^{\delta}\in Y$.

Proof.

We may estimate

[TABLE]

which proves the desired result. ∎

2.2 Error estimates

Convergence results for convex regularisation are well known. We state some standard results [7, 33]: we assume henceforth a regularity condition on $x^{\dagger}$; namely, that it fulfills the following source condition:

[TABLE]

Subsequently, $\xi:=A^{\ast}w$ is the subgradient of $\mathcal{R}$ at $x^{\dagger}$. We remark that much of the analysis (concerning the convergence results, not the rates) below is valid with (10) replaced by weaker conditions, e.g., in form of variational source conditions [16, 8, 9]. We have the following error estimates:

Proposition 2**.**

Let $x^{\dagger}$ satisfy the source condition (10). Then

[TABLE]

and

[TABLE]

for all $\alpha\in(0,\alpha_{\text{max}})$ and $y,y^{\delta}\in Y$.

Note also that

[TABLE]

for all $\alpha\in(0,\alpha_{\text{max}})$ and $y^{\delta}\in Y$. The inequalities (11), of course, holds analogously for the noise-free variables. Note that from (10) and (2), we obtain the following estimate:

[TABLE]

cf. [18]. We will utilise this in the convergence proofs to come in the latter sections.

2.3 The heuristic discrepancy rule

In terms of the residual variables, the heuristic discrepancy functional may be expressed as

[TABLE]

and $\alpha_{\ast}$ is selected as its minimiser. In the paper [18], the following error estimate was derived:

Theorem 1**.**

Let $x^{\dagger}$ satisfy the source condition (10), let $\alpha_{\ast}$ be chosen according to the heuristic discrepancy rule and suppose that $\delta^{\ast}:=\|Ax^{\delta}_{\alpha}-y^{\delta}\|\neq 0$. Then there exists a constant $C>0$ such that

[TABLE]

The above estimate is of restricted utility as one has no control over the value of $\delta^{\ast}$. If one assumes the condition

[TABLE]

where $Q:Y\to(\overline{\text{range}(A)})^{\perp}$ is an orthogonal projection and $\varepsilon>0$, which was introduced by Hanke and Raus in [14], then one can prove that

[TABLE]

thus without the troublesome prefactor and using (13) and (14), it is possible to prove convergence of the regularisation method, i.e.,

[TABLE]

The condition (13) is quite abstract and if one considers the case in which $\overline{\text{range}(A)}=Y$, it would follow that $Q:Y\to\emptyset$ and consequently $\|Q(y-y^{\delta})\|=0\geq\varepsilon\|y-y^{\delta}\|$, i.e., the condition is not satisfied.

Another noise condition was subsequently postulated in [18]; namely, if there exists $\varepsilon\in(0,1)$ such that $y^{\delta}-y\in\mathcal{N}$, where

[TABLE]

then, if $y^{\delta}-y\in\mathcal{N}$, it would follow that

[TABLE]

from which it is again possible to prove convergence. It still remains unclear when (16) is satisfied. It is actually the main aim of this paper to replace these conditions with more practical ones.

For heuristic rules, it is often standard to show convergence of the selected parameter as the noise level tends to zero:

Proposition 3**.**

Let $\alpha_{\ast}$ be the minimiser of the heuristic discrepancy functional. Then $\alpha_{\ast}\to 0$ as $\delta\to 0$.

For the proof, we refer to [18]. In order to prove convergence, the most difficult part is to derive a condition that prevents $\alpha_{\ast}$ from decaying too rapidly. This always involves some restriction on the noise. In the next theorem, we impose such a noise restriction in the form of an auto-regularisation condition:

Theorem 2**.**

Let the source condition (10) be satisfied, $\alpha_{\ast}$ be selected according to the heuristic discrepancy rule, $\delta^{\ast}\neq 0$ and assume the auto-regularisation condition

[TABLE]

holds for all $\alpha\in(0,\alpha_{\text{max}})$ and $y^{\delta}\in Y$. Then it follows that the method converges; i.e.,

[TABLE]

as $\delta\to 0$.

Proof.

From the estimate (12), one may then immediately estimate the data propagation error courtesy of the auto-regularisation condition as

[TABLE]

Since $\alpha_{\ast}$ minimises the heuristic discrepancy functional, it follows that

[TABLE]

Hence, one may choose $\alpha=\alpha(\delta)$ such that $\alpha(\delta)\to 0$ as $\delta^{2}/\alpha(\delta)\to 0$ for $\delta\to 0$. Furthermore,

[TABLE]

Thus, we may conclude that

[TABLE]

For the approximation error, it follows that

[TABLE]

Hence each term in (12) tends to 0 as $\delta\to 0$. ∎

From Lemma (1), it suffices for (18) to show that there exists $C>0$ such that

[TABLE]

Obviously, it is then enough to prove

[TABLE]

for some positive constant $C$.

The auto-regularisation condition is an implicit condition on the noise. One may observe that it resembles the condition of [21] in the linear case. Certainly, in this form, it is still an abstract condition, although we will reduce it in Section (3) to a more reasonable form from which we can extract more understanding.

2.3.1 Convergence rates

With the aid of the source condition, auto-regularisation condition and an additional regularity condition, we can even derive rates of convergence. We start with the following proposition:

Proposition 4**.**

Suppose that $\partial\mathcal{R}^{\ast}:X^{\ast}\to X^{\ast\ast}$ is continuous at [math]. Then, for all ${y^{\delta}}$ with $\|{y^{\delta}}\|\geq C$, there is a constant such that

[TABLE]

Proof.

As $\langle{p_{\alpha}^{\delta}},{y^{\delta}}\rangle\geq 0$, we have

[TABLE]

Now assume that (21) does not hold. Then there is a sequence with $\|{y^{\delta}}\|\geq C$ and $\|\frac{{p_{\alpha}^{\delta}}}{\alpha}\|\to 0$. However, this implies that $A^{\ast}\frac{{p_{\alpha}^{\delta}}}{\alpha}\to 0$ in $X^{\ast}$ and by continuity, also that

[TABLE]

However, this leads to a contradiction as then

[TABLE]

Thus (21) must hold. ∎

We now state the main convergence rates result for the HD rule:

Proposition 5**.**

Let the source condition (10) hold, $\alpha_{\ast}$ be selected according to the heuristic discrepancy rule and suppose the auto-regularisation condition (18) is satisfied. Assume, in addition, that $\|{y^{\delta}}\|\geq C$ and that $\partial\mathcal{R}^{*}:X^{\ast}\to X^{\ast\ast}$ is continuous at [math]. Then

[TABLE]

for $\delta>0$ sufficiently small.

Proof.

Note that from Proposition (4) and since $\alpha_{\ast}$ is the global minimiser, for any $\alpha$, we have that $\alpha_{\ast}\leq C\psi_{\text{HD}}(\alpha_{\ast},y^{\delta})\leq C\psi_{\text{HD}}(\alpha,y^{\delta})$. Observe that from (12), it follows that

[TABLE]

choosing $\alpha=\alpha(\delta)=\delta$. ∎

2.4 The Hanke-Raus rule

As with the HD rule, the Hanke-Raus functional may be reexpressed as

[TABLE]

To the best of the authors’ knowledge, in contrast to the heuristic discrepancy rule, the Hanke-Raus rule has not yet been rigorously analysed in the convex variational setting although it has been tested numerically for total variation regularisation (cf. [24]).

Note that the Hanke-Raus functional is not expressed in terms of a norm, thus there is no a-priori guarantee that it remains positive (as it is in the linear case). We therefore provide the following proposition:

Proposition 6**.**

We have that

[TABLE]

for all $\alpha\in(0,\alpha_{\text{max}})$ and $y^{\delta}\in Y$.

Proof.

We obtain that

[TABLE]

for all $\alpha\in(0,\alpha_{\text{max}})$, which follows from (5). ∎

Proposition 7**.**

We have that

[TABLE]

for all $\alpha\in(0,\alpha_{\text{max}})$ and $y^{\delta}\in Y$.

Proof.

We can estimate

[TABLE]

for all $\alpha\in(0,\alpha_{\text{max}})$, where we have used (5). ∎

Proposition 8**.**

Let $\alpha_{\ast}$ be the minimiser of the Hanke-Raus functional. Then $\alpha_{\ast}\to 0$ as $\delta\to 0$.

Proof.

Since $\psi_{\text{HR}}(\alpha_{\ast},y^{\delta})\leq\psi_{\text{HD}}(\alpha_{\ast},y^{\delta})\to 0$ as $\delta\to 0$, we deduce that $\|Ax^{\delta}_{\alpha_{\ast}}-y^{\delta}\|\to 0$ as $\delta\to 0$. Then the proof is the same as the one given in [18] for $\alpha_{\ast}$ selected according to the heuristic discrepancy rule. ∎

Next we state the main convergence theorem for the Hanke-Raus rule for which we again require an auto-regularisation condition:

Theorem 3**.**

Let the source condition (10) be satisfied and let $\alpha_{\ast}$ be the minimiser of $\psi_{HR}(\alpha,y^{\delta})$ and suppose that there exists a positive constant $C>0$ such that

[TABLE]

for all $\alpha\in(0,\alpha_{\text{max}})$ and $y^{\delta}\in Y$. Then

[TABLE]

Proof.

Let $\alpha_{\ast}$ be the minimiser of the Hanke-Raus functional. Then, as before, we estimate the Bregman distance as (12) and from (22), deduce that

[TABLE]

Now, notice that the last two terms can be estimated as

[TABLE]

Moreover,

[TABLE]

as $\delta\to 0$, and obviously

[TABLE]

choosing $\alpha$ such that $\alpha\to 0$ and $\delta^{2}/\alpha\to 0$ as $\delta\to 0$. The result then follows from the fact that the other terms in (12) also vanish. ∎

By Lemma (1), sufficient for the auto-regularisation condition (8) is that

[TABLE]

2.4.1 Convergence rates

The following convergence rates theorem for the Hanke-Raus rule requires an additional condition, which we, however, will not investigate further as it is beyond the scope of our work.

Proposition 9**.**

Let the source condition (10) hold, $\alpha_{\ast}$ be selected according to the Hanke-Raus rule and supposes the auto-regularisation condition (22) is satisfied. Assume, in addition that $\|{y^{\delta}}\|\geq C$ for all $\delta$ and suppose that a constant exists with

[TABLE]

Then

[TABLE]

for $\delta>0$ sufficiently small.

Proof.

Notice that the imposed conditions imply that $\alpha_{\ast}\leq C\psi_{\text{HR}}(\alpha_{\ast},y^{\delta})$. Now,

[TABLE]

for $\delta$ sufficiently small. ∎

2.5 The quasi-optimality rules

The principle behind the quasi-optimality rule is to minimise the difference of two successive approximations of the solution. In the linear case, the difference is measured with the norm, but in the convex setting, we use the Bregman distance. Therefore, possibilities for the quasi-optimality rule include choosing $\alpha_{\ast}$ as the minimiser of $D^{\text{sym}}_{\xi^{II}_{\alpha,\delta},\xi^{\delta}_{\alpha}}(x^{II}_{\alpha,\delta},x^{\delta}_{\alpha})$, $D_{\xi^{II}_{\alpha,\delta}}(x^{\delta}_{\alpha},x^{II}_{\alpha,\delta})$ or $D_{\xi^{\delta}_{\alpha}}(x^{II}_{\alpha,\delta},x^{\delta}_{\alpha})$.

Similarly as for the Hanke-Raus rule, the authors are not aware of any other analysis of the quasi-optimality rule defined as above. The discrete version was considered in [18]. There, the noise condition postulated was a generalisation into the convex setting of the auto-regularisation set of [10], and the numerical performance of the rule with $D_{\xi^{II}_{\alpha,\delta}}(x^{\delta}_{\alpha},x^{II}_{\alpha,\delta})$ was tested in [24]. The performance of the latter rule in the aforementioned reference and also in our own numerical experiments proved to be quite poor and therefore we omit it. The rule with the symmetric Bregman distance performs reasonably, on the other hand. The remaining version is generally the best performing of all the quasi-optimality rules. More light will be shed on this, however, in the numerics section.

2.5.1 The symmetric quasi-optimality rule

Similarly to the previous two rules discussed, the symmetric quasi-optimality functional may also be expressed in terms of residuals:

Proposition 10**.**

We have that

[TABLE]

for all $\alpha\in(0,\alpha_{\text{max}})$ and $y^{\delta}\in Y$.

Proof.

We have

[TABLE]

for all $\alpha\in(0,\alpha_{\text{max}})$ and $y^{\delta}\in Y$, which is what we wanted to show. ∎

Proposition 11**.**

We have

[TABLE]

for all $\alpha\in(0,\alpha_{\text{max}})$ and $y^{\delta}\in Y$.

Proof.

It follows trivially from the observation that

[TABLE]

for all $\alpha\in(0,\alpha_{\text{max}})$ and $y^{\delta}\in Y$. ∎

Proposition 12**.**

Let $\alpha_{\ast}$ be the minimiser of the symmetric quasi-optimality functional. Then $\alpha_{\ast}\to 0$ as $\delta\to 0$.

Proof.

Since $\psi_{\text{SQO}}(\alpha_{\ast},y^{\delta})\leq\psi_{\text{HR}}(\alpha_{\ast},y^{\delta})\leq\psi_{\text{HD}}(\alpha_{\ast},y^{\delta})$, the proof that $\alpha_{\ast}\to 0$ as $\delta\to 0$ is identical to the one given in [18] for the heuristic discrepancy rule. ∎

Similar as above, an auto-regularisation condition leads to convergence:

Theorem 4**.**

Let the source condition (10) be satisfied and, $\alpha_{\ast}$ be the minimiser of $\psi_{\text{SQO}}(\alpha,y^{\delta})$ and suppose that

[TABLE]

for all $\alpha\in(0,\alpha_{\text{max}})$ and $y,y^{\delta}\in Y$.

Then

[TABLE]

Proof.

From (12) and (24), it remains to estimate

[TABLE]

Now, the last two “remainder”terms can be estimated from above by

[TABLE]

Moreover,

[TABLE]

and

[TABLE]

as $\delta\to 0$ for $\alpha$ chosen appropriately as before. The result then follows. ∎

2.5.2 Convergence rates

For completeness, we provide convergence rates results:

Proposition 13**.**

Let the source condition (10) hold, $\alpha_{\ast}$ be selected according to the symmetric quasi-optimality rule and suppose the auto-regularisation condition (24) is satisfied. Assume, that $\|{y^{\delta}}\|\geq C$ for all $\delta$ sufficiently small and in addition, that

[TABLE]

Then

[TABLE]

for $\delta>0$ sufficiently small.

Proof.

We have

[TABLE]

where we used that $\alpha_{\ast}\leq\psi_{\text{SQO}}(\alpha_{\ast},y^{\delta})$, which follows from (25). ∎

2.5.3 The right quasi-optimality rule

As the expression for the right quasi-optimality functional contains the cumbersome $\mathcal{R}$-functional terms, estimates in which they do not appear may be of utility:

Proposition 14**.**

There exists a positive constant $C$ such that

[TABLE]

for all $\alpha\in(0,\alpha_{\text{max}})$ and $y^{\delta}\in Y$.

Proof.

We may write

[TABLE]

From the optimality of $x^{II}_{\alpha,\delta}$, we have that

[TABLE]

i.e.,

[TABLE]

from which we get that

[TABLE]

and the desired estimate from above subsequently follows. ∎

Proposition 15**.**

Let $\alpha_{\ast}$ be selected according to the right quasi-optimality rule. Then $\alpha_{\ast}\to 0$ as $\delta\to 0$ for all $y^{\delta}\in Y$.

Proof.

Since we can estimate the quasi-optimality functional by the heuristic discrepancy functional, as per the previous proposition, the result follows from [18]. ∎

Theorem 5**.**

Let $x^{\dagger}$ satisfy (10), $\alpha_{\ast}$ be selected according to the right quasi-optimality rule, and suppose that there exists a positive constant $C>0$ such that

[TABLE]

holds for all $\alpha\in(0,\alpha_{\text{max}})$ and $y,y^{\delta}\in Y$. Then

[TABLE]

as $\delta\to 0$.

We omit the proof as it is analogous to the above. Note that if the source condition (10) holds, $\alpha_{\ast}$ is selected according to the right quasi-optimality rule and the auto-regularisation condition (27) is satisfied, then one may also prove that

[TABLE]

for $\delta>0$ sufficiently small, provided that $\alpha_{\ast}\leq\psi_{\text{RQO}}(\alpha_{\ast},y^{\delta})$.

Remark**.**

Observe that the heuristic discrepancy, the Hanke-Raus, and the symmetric quasi-optimality rules can all be expressed in terms of the residuals of the Bregman iteration ${p_{\alpha}^{\delta}},{p_{\alpha,\delta}^{II}}$. It should be noted that in the linear case they can all be subsumed under the so-called family of R1-rules [32]. The similarity of patterns in the formulas for $\psi$ may provide a hint that such a larger family of rules could be defined in the convex case as well.

3 Diagonal operator

In the following analysis, we consider the case in which the operator $A:X\to Y$ is diagonal between spaces of summable sequences; in particular, where $X=\ell^{q}(\mathbb{N})$ and $Y=\ell^{2}(\mathbb{N})$ and the regularisation functional is selected as the $\ell^{q}$-norm to the $q$-th power. The main objective in this setting is to further investigate the auto-regularisation conditions and to illustrate their validity for specific instances.

Let $\{e_{n}\}_{n\in\mathbb{N}}$ be a basis for $X$ and let $\{\lambda_{n}\}_{n\in\mathbb{N}}$ be a sequence of real scalars monotonically decaying to [math]. Then we define a diagonal operator $A:\ell^{q}(\mathbb{N})\to\ell^{2}(\mathbb{N})$,

[TABLE]

The regularisation functional is chosen as

[TABLE]

In this situation, the operator decouples and the components of the regularised solution can be computed independently of each other. Thus, for notational purposes, we opt to omit the sequence index $n$ for the components of the regularised solutions and write

[TABLE]

where ${\mathpzc{x}_{\alpha}},{\mathpzc{x}_{\alpha}^{\delta}},\mathpzc{y}^{\delta},\mathpzc{y}\in\mathbb{R}$. As the problem decouples, ${\mathpzc{x}_{\alpha}}$ and ${\mathpzc{x}_{\alpha}^{\delta}}$ can be computed by an optimisation problem on $\mathbb{R}$, i.e., the optimality conditions lead to

[TABLE]

for all $\alpha\in(0,\alpha_{\text{max}})$ and $\mathpzc{y}^{\delta}\in\mathbb{R}$. Here $h_{q,\gamma}$ is the inverse of the function $h_{q,\gamma}:\mathbb{R}\to\mathbb{R}$,

[TABLE]

Note that $h_{q,\gamma_{n}}^{-1}$ corresponds to a proximal operator on $\mathbb{R}$.

Define ${\mathpzc{p}^{\delta}_{\alpha}}:=\mathpzc{y}^{\delta}-\lambda_{n}{\mathpzc{x}_{\alpha}^{\delta}}$ and ${\mathpzc{p}_{\alpha}}$ analogously in case of exact data; furthermore we use the expressions $\mathpzc{y}$, $\mathpzc{y}^{\delta}$, ${\Delta\mathpzc{y}}$, ${\Delta\mathpzc{p}_{\alpha}}$, $\Delta\mathpzc{p}^{II}_{\alpha}$ to denote the components of ${y^{\delta}},$ $y,$ ${\Delta y},$ ${\Delta p_{\alpha}}$, $\Delta p^{II}_{\alpha}$, respectively, where we again omit the sequence index $n$ in the notation:

[TABLE]

For the $\ell^{q}$-case we can apply an appropriate scaling to reduce the corresponding inequalities: in fact, letting

[TABLE]

we obtain

[TABLE]

Note that for $x>0$ we have

[TABLE]

We now state useful estimates for the function $h_{q}^{-1}$:

Lemma 2**.**

For $1<q<\infty$, $q\not=2$, there exist constants $\underline{D}_{p},D_{q},$ and for any $\tau>0$, a constant $D_{q,\tau}$ such that for all $x_{1}>0$ and $|x_{2}|\leq x_{1}$,

[TABLE]

Proof.

For any $z_{1}>0,|z_{2}|\leq z_{1}$, we have

[TABLE]

where

[TABLE]

Replacing $z_{i}$ by $h_{q}^{-1}(x_{i})$ yields the lower bound and the first upper bound in (33). In case $1<q<2$, we find that

[TABLE]

where we used that $h_{q}^{-1}(x_{1})^{2-q}\geq 0$ in the denominator and the estimate $h_{q}^{-1}(x_{1})\leq x_{1}^{\frac{1}{q-1}}$ that follows from (32). Now consider the case $q>2$. Then

[TABLE]

where we used the estimate

[TABLE]

This yields the result. ∎

3.0.1 The heuristic discrepancy rule

In case the forward operator is diagonal, we may reduce the auto-regularisation condition to Muckenhoupt-type inequalities [26] which may shed more light on them.

Proposition 16**.**

Let $A:\ell^{q}(\mathbb{N})\to\ell^{2}(\mathbb{N})$ be a diagonal operator and let $\mathcal{R}=\frac{1}{q}\|\cdot\|^{q}_{\ell^{q}}$. If

[TABLE]

is satisfied, where,

[TABLE]

Then there exists a positive constant $C>0$ such that

[TABLE]

for all $\alpha\in(0,\alpha_{\text{max}})$ and $y,y^{\delta}\in Y$.

Proof.

For $A:\ell^{q}(\mathbb{N})\to\ell^{2}(\mathbb{N})$ a diagonal operator, (20) may be rewritten as

[TABLE]

Now, we may decompose the above sum as the superposition of two sums to obtain

[TABLE]

Thus we may obtain the desired inequality. ∎

We are now in the position to state the main convergence result for the heuristic discrepancy principle.

Theorem 6**.**

Let $A$ and $\mathcal{R}$ be as in Proposition (16) and $q\in(1,\infty)$. Let $\alpha_{*}$ be selected by the heuristic discrepancy rule, let the source condition (10) be satisfied, $\delta^{\ast}\neq 0$ and suppose that there are constants $C_{1}$, $C_{2}$ such that for all ${y^{\delta}}$ it holds that

[TABLE]

Then

[TABLE]

Proof.

From the preceding results, it remains to verify that (36) implies (34). For this, we estimate the ratio $\frac{{\Delta\mathpzc{p}_{\alpha}}}{{\Delta\mathpzc{y}}}$. We note that by monotonicity, this expression is always positive, and moreover, is invariant when $\mathpzc{y}$, $\mathpzc{y}^{\delta}$ are switched and when $\mathpzc{y}$, $\mathpzc{y}^{\delta}$ are replaced by $-\mathpzc{y}$, $-\mathpzc{y}^{\delta}$. Thus, without loss of generality we may assume that $\mathpzc{y}>0$ and $|\mathpzc{y}^{\delta}|\leq\mathpzc{y}$. Using (29), Lemma (2) with $x_{1}=\frac{\mathpzc{y}}{\lambda_{n}\eta_{n}}=\frac{\max\{|\mathpzc{y}|,|\mathpzc{y}^{\delta}|\}}{\lambda_{n}\eta_{n}}$ and $x_{2}=\frac{\mathpzc{y}^{\delta}}{\lambda_{n}\eta_{n}}$, we find that

[TABLE]

A brief consideration yields that (34) is satisfied if the corresponding inequality is satisfied when $I_{\text{HD}}$ is replaced with any $I_{\text{HD}}^{\prime}\subset I_{\text{HD}}$ (on the left and right-hand side of the inequality). Thus, a sufficient inequality is when the index set

[TABLE]

is used in place of $I_{\text{HD}}$. The condition in this index set can be simplified to

[TABLE]

Note that $\lambda_{n}\eta_{n}=\left(\frac{\alpha}{\lambda^{q}}\right)^{\frac{1}{2-q}}$. Thus, in any case, we obtain that

[TABLE]

with $C_{q}=\left[h_{q^{*}}\left(\left(\frac{C-1}{\underline{D}_{p}}\right)^{\frac{1}{q^{*}-2}}\right)\right]^{2-q}$. In the next step, we observe that Lemma (2) yields the upper bound

[TABLE]

in case that $1\leq q^{*}\leq 2$. The same estimate holds for $q^{*}>2$ whenever $\frac{\max\{|\mathpzc{y}|,|\mathpzc{y}^{\delta}|\}}{\lambda_{n}\eta_{n}}\geq\tau$. However, as the left-hand side is a sum over the complement of ${I_{\text{HD}}^{\prime}}$, the latter condition holds at the complement; hence (38) is true in any case. As $\frac{2-q^{*}}{q^{*}-1}=q-2$, and since the condition with an upper bound for the left-hand side is sufficient for (34), it is thus shown that (36) implies (34).

∎

3.0.2 The Hanke-Raus rule

The following are sufficient conditions for the auto-regularisation condition of the Hanke-Raus rule to hold in the present setting.

Proposition 17**.**

Let $A:\ell^{q}(\mathbb{N})\to\ell^{2}(\mathbb{N})$ be a diagonal operator and let $\mathcal{R}=\frac{1}{q}\|\cdot\|^{q}_{\ell^{q}}$. Suppose that for all $\mathpzc{y},\mathpzc{y}^{\delta}$ and all $n\in\mathbb{N}$,

[TABLE]

Furthermore, let the condition

[TABLE]

hold, where

[TABLE]

Then there exists a positive constant $C>0$ such that

[TABLE]

for all $\alpha\in(0,\alpha_{\text{max}})$.

Proof.

Recall that it suffices to prove that

[TABLE]

We may define

[TABLE]

and observe that this quantity is in any case nonegative and for $n\in I_{\text{HR}}$ larger than $\theta>\frac{1}{2}$. Thus,

[TABLE]

Therefore, (41) holds for $n\in I_{\text{HR}}$. The remaining sum can be bounded by (40) and because of (39), the sum over $I_{\text{HR}}^{c}$ on the right-hand side can be estimated by [math] from below. This suffices to prove the statement. ∎

We proceed by verifying (39). Contrary to the heuristic discrepancy case, we have to impose a restriction on the regularisation functional $\mathcal{R}=\frac{1}{q}\|\cdot\|^{q}_{\ell^{q}}$.

Lemma 3**.**

If $q\geq\frac{3}{2}$, then it follows that

[TABLE]

for all $\alpha\in(0,\alpha_{\text{max}})$ and $y^{\delta}\in Y$.

Proof.

Setting $z_{1}=\frac{{y^{\delta}}}{\eta_{n}\lambda_{n}}$, $z_{2}=\frac{y}{\eta_{n}\lambda_{n}}$, the statement is verified if we can show that for any $z_{1},z_{2}$,

[TABLE]

Since $h_{q^{*}}^{-1}\left(z_{1}+h_{q^{*}}^{-1}(z_{1})\right)=h_{q^{*}}^{-1}\left(h_{q^{*}}(h_{q^{*}}^{-1}(z_{1}))+h_{q^{*}}^{-1}(z_{1})\right)$, it is enough to show that the mapping (note the one-to-one correspondence of $z_{i}\mapsto h_{q^{*}}^{-1}(z_{i})$),

[TABLE]

is monotonically increasing. As this function is differentiable everywhere except at $p=0$, it suffices to prove the inequality

[TABLE]

for any $p\in\mathbb{R}$. Since $F$ is antisymmetric and hence $F^{\prime}$ is symmetric, it is sufficient to prove this inequality for $p>0$. Setting $r=h_{q^{*}}^{-1}\left(h_{q^{*}}(p)+p\right)$, we thus have to show that

[TABLE]

which reduces to

[TABLE]

Some algebraic manipulation allows us to express $p^{q-2}$ in terms of $\zeta:=\frac{r}{p}$ as

[TABLE]

The right-hand side is monotonically decreasing for $\zeta\in[1,2]$, thus we may invert the equation and express $\zeta$ as function of $p$. Setting $r^{q^{*}-2}=p^{q^{*}-2}\zeta^{q^{*}-2}$, the inequality we would like to prove is then

[TABLE]

For $q^{*}\leq 2$, the left-hand side is clearly negative for $\zeta\in[1,2]$, and thus the inequality is trivially satisfied. For $q^{*}>2$, it can be verified by a detailed analysis that for $q^{*}-1\leq 2$,

[TABLE]

which proves (42). Note that the condition $q^{*}\leq 3$ is equivalent to $q\geq\frac{3}{2}$. ∎

Theorem 7**.**

Let $A$ and $\mathcal{R}$ be as in Proposition (16) with $\frac{3}{2}\leq q<\infty$. Let $\alpha_{*}$ be selected by the Hanke-Raus functional, let the source condition (10) be satisfied, $\delta^{\ast}\neq 0$ and suppose that there is a constant $C_{1}$, which is sufficiently small and a constant $C_{2}$ such that for all ${y^{\delta}}$, it holds that

[TABLE]

Then

[TABLE]

Proof.

We verify that (43) implies (40). Again with similar considerations as before we may assume $|\mathpzc{y}^{\delta}|\leq\mathpzc{y}$ and use

[TABLE]

Then

[TABLE]

As before we may define a new index set $I_{\text{HR}}^{\prime}$ of indices, where

[TABLE]

such that

[TABLE]

with $C_{q}=\left[h_{q^{*}}\left(\left(\frac{\frac{1}{\theta}-1}{\underline{D}_{p}}\right)^{\frac{1}{q^{*}-2}}\right)\right]^{2-q}$. Since $0\leq{\mathpzc{p}_{\alpha}}\leq|\mathpzc{y}|$ (and similar for the $\delta$-part), we obtain

[TABLE]

which yields a subset

[TABLE]

where $C_{q}^{\prime}=\max\{2^{2-q},1\}C_{q}$. Note that the restriction $\theta>\frac{1}{2}$ can always be achieved by postulating that $C_{q}^{\prime}$ is sufficiently small. We observe that except for the constant $C_{q}^{\prime}$, $I_{\text{HR}}^{\prime\prime}$ agrees with $I^{\prime}_{\text{HD}}$; hence we may estimate the right-hand side from below by $C\sum_{n\in I_{\text{HR}}^{\prime\prime}}|{\Delta\mathpzc{y}}|^{2}$. On the other hand, the left-hand side can be bounded from above exactly the same as for the heuristic discrepancy rule, which shows the sufficiency of (43) for (40). ∎

We remark that apart from the smallness condition on $C_{1}$, this is the same condition as for the heuristic discrepancy rule. This completely agrees with the linear theory ($q=2$).

3.0.3 The symmetric quasi-optimality rule

Proposition 18**.**

Let $A:\ell^{q}(\mathbb{N})\to\ell^{2}(\mathbb{N})$ be a diagonal operator and let $\mathcal{R}=\frac{1}{q}\|\cdot\|^{q}_{\ell^{q}}$. Suppose that for all $\mathpzc{y},\mathpzc{y}^{\delta}$ and all $n\in\mathbb{N}$

[TABLE]

Furthermore, let the condition

[TABLE]

hold, where for some constant $C$

[TABLE]

Then there exists a constant such that

[TABLE]

for all $\alpha\in(0,\alpha_{\text{max}})$ and $y^{\delta}\in Y$.

Proof.

By Lemma (1), it suffices to show that

[TABLE]

By the nonnegativity of (44) and monotonicity, we have

[TABLE]

and

[TABLE]

such that by the definition of $I_{\text{SQO}}$ the sum over $I_{\text{SQO}}$ on the left-hand side of (46) can be bounded by that on the right-hand side. The sum over the complement of $I_{\text{SQO}}$ on the right can be bounded by (44) from below by [math] and the corresponding sum on the left is bounded by condition (45), which proves the statement. ∎

Similar as for the Hanke-Raus rule, we first have to verify the nonnegativity of certain expressions:

Lemma 4**.**

If $q\geq\frac{3}{2}$, then

[TABLE]

for all $\alpha\in(0,\alpha_{\text{max}})$ and $\mathpzc{y},\mathpzc{y}^{\delta}\in\mathbb{R}$.

Proof.

Recall the mapping $F:{\mathpzc{p}_{\alpha}}\mapsto{\mathpzc{p}^{II}_{\alpha}}$ defined above. In order to prove monotonicity, it is enough to show that

[TABLE]

or

[TABLE]

Thus, if $F$ would be monotone and Lipschitz continuous, then the inequality would hold. Thus, we require the condition that

[TABLE]

In the notation of above, this means that

[TABLE]

In terms of $\zeta\in[1,2]$ this means that, in addition to the condition for the positivity of the Hanke-Raus rule, we require that

[TABLE]

In particular, this inequality is satisfied for any $q^{*}\geq 1$, which shows the result. ∎

Theorem 8**.**

Let $A$ and $\mathcal{R}$ be as in Proposition (18) with $\frac{3}{2}\leq q<\infty$. Let $\alpha_{*}$ be selected by the symmetric quasi-optimality rule, let the source condition (10) be satisfied, $\delta^{\ast}\neq 0$ and suppose that there is a constant $C_{1}$, which is sufficiently small and constants $C_{2},C_{3}$ such that for all $\mathpzc{y}^{\delta}$, it holds that

[TABLE]

where for some constant $C_{3}$,

[TABLE]

Then

[TABLE]

Proof.

Similar as for the Hanke-Raus rule, the second condition in the definition of $I_{\text{SQO}}$,

[TABLE]

holds true if

[TABLE]

The sum over this set on the left-hand side of (45) can be bounded in the same way as before by the first sum in (47). The second sum is an upper bound for the sum on the left-hand side in the complement of $I_{\text{SQO}}$, where (48) is not satisfied. On $I_{\text{SQO}}$, we have the estimate that

[TABLE]

where we used (37) in the estimate where $C$ appears and in the last step a bound for $z>0$ on $I_{\text{SQO}}$ of the form

[TABLE]

that can be obtained by similar means as above. ∎

Remark**.**

The condition in (47) has an additional sum over the index set $I_{HR}^{\prime\prime}\cap I_{2}^{c}$ on the left-hand side. It might be possible to prove that this set is empty, e.g., if $I_{HR}^{\prime\prime}\subset I_{2}$. Then the corresponding sum would vanish, and this happens in the linear case ($q=2$). However, we postpone a more detailed analysis of this issue to the future.

We also point out that the Muckenhoupt-type conditions (36), (43), and (47) (except for the additional sum) agree with the respective ones for the linear case $q=2$ so that they appear, in fact, as natural extensions of the linear convergence theory.

3.1 Case study of noise restrictions

For the cases that the operator ill-posedness, the regularity of the exact solution and the noise show some typical behaviour, we investigate the restrictions that the Muckenhoupt-type conditions (43) and (36) impose on the noise. In particular, we would like to point out that the restrictions are not at all unrealistic and they are satisfied in paradigmatic situations.

Consider a polynomially ill-posed problem,

[TABLE]

where the exact data have a higher decay compared to $\lambda_{n}$ as a result of the regularity of the exact solution,

[TABLE]

We furthermore assume a standard polynomial decay of the error terms:

[TABLE]

The restriction $\kappa<\nu$ is natural as the noise is usually less regular than the exact solution. In the linear case, Muckenhoupt-type conditions lead to restrictions on the regularity of the noise, i.e., upper bounds for the decay rate $\kappa$. This is perfectly in line with their interpretation as conditions for sufficiently irregular noise.

In the following we write $\sim$ if the left and right expressions can be estimated by constants independent of $n$. (There might be a $q$-dependence, however).

Let us define the following weight that appears in (43) and (36):

[TABLE]

We additionally impose the restriction that for sufficiently large $n$, $W_{n}\to 0$ monotonically. If $2-q>0$, this is trivially satisfied, while for $2-q<0$, we require that

[TABLE]

Under these assumptions, for any $\alpha$ sufficiently small, we find an ${n^{*}}$ such that $W_{{n^{*}}}=C_{1}\alpha$ and $W_{n}\leq C_{1}\alpha$ for $n\geq{n^{*}}$. Expressing $\alpha$ in terms of $W_{{n^{*}}}$ yields a sufficient condition for (43), (36) as

[TABLE]

By the straightforward estimate $\max\{1,|1+\frac{s_{n}\delta}{C_{2}}n^{\nu-\kappa}|\}\sim 1+\delta n^{\nu-\kappa}$, the inequality (50) reduces to

[TABLE]

For any $x\geq 0$ and $0\leq z\leq 1$, it holds that

[TABLE]

We use this inequality with $z=\frac{n}{n^{*}}$ and $x=\delta{n^{*}}$. Then, we obtain the sufficient conditions

[TABLE]

These inequalities are satisfied if the exponent for $n$ is strictly larger than $-1$. This finally leads to the restrictions

[TABLE]

Note that for $q>2$, we additionally require (49).

We hope to have the reader convinced that the imposed conditions on the noise are not too restrictive and, in particular, the set of noise that satisfies them is nonempty. These conditions provide a hint for which cases the methods may work or fail:

In case $\frac{3}{2}\leq q\leq 2$, both the heuristic discrepancy and the Hanke-Raus are reasonable rules. The conditions on the noise are less restrictive the smaller $q$ is.

In case $1<q<\frac{3}{2}$, our convergence analysis only applies to the heuristic discrepancy rule, as the nonnegativity condition of the Hanke-Raus rule is not satisfied in this case. It could be said that the heuristic discrepancy rule is the more robust one then.

In case $q>2$, we observe that the restriction on the noise depends on the regularity of the exact solution. For highly regular exact solutions ($\nu\gg 1$) the noise condition might fail to be satisfied as $q$ becomes very large. This happens for both the heuristic discrepancy, and the Hanke-Raus rules.

We did not include the quasi-optimality condition in this analysis as this still requires further analysis. However, the conditions for it are usually even more restrictive than for the Hanke-Raus rules and we expect similar problems for the case $q>2.$

4 Numerical experiments

In this section, we would like to numerically illustrate and verify the theoretical findings of the preceding sections. It should be stressed that the preceding convergence analysis provides an important piece in understanding the behaivour, but there exist further factors that influence the actual quality of the results using heuristic rules (e.g., for optimal-order results, a regularity condition on $x^{\dagger}$ is often required and the value of the constants in the estimates are important). It should be noted as well that our results only proved sufficient conditions for convergence and they do not say when a certain method may fail. (For instance, by including $\delta$-dependent estimates, one may find weaker convergence conditions that hold for certain ranges of the noise level). Still, a preliminary understanding can be gained from this. To further illustrate the behaviour, we perform numerical experiments that we describe in this section.

In all experiments, we consider $\mathcal{R}=\frac{1}{q}\|\cdot\|^{q}_{\ell^{q}}$ in the discretised space $\mathbb{R}^{n}$. Due to numerical errors resulting from the discretisation, we opt to choose the parameter $\alpha_{\ast}\in[\alpha_{\text{min}},\alpha_{\text{max}}]$. We also choose to select $\alpha_{\text{max}}=\|A\|^{2}$ (apart from for TV regularisation, which we will define) and for the more tricky issue of the lower bound, we set $\alpha_{\text{min}}=\sigma_{\text{min}}$, the smallest singular value of $A^{\ast}A$. Other methodologies for selecting $\alpha_{\text{min}}$ were suggested in [22, 12]. Because of several effects, it may happen that the heuristic functionals exhibit multiple local minima and selecting $\alpha_{\ast}$ is no longer an obvious task. For this reason, we select $\alpha_{\ast}$ as the interior global minimum within the aforementioned interval. Additionally, we always rescale the forward operator and exact solution so that $\|A\|=\|x^{\dagger}\|=1$. In each experiment, we compute 10 different realisations of the noise in which the noise level is logarithmically increasing. We also compute the error (the measure for which will differ for the various regularisation schemes) induced by each parameter choice rule, as well as the optimal parameter choice, which will be computed as the minimiser of the respective error functional itself.

For our operators, we use the tomography operator (tomo) from Hansen’s tools (cf. [15]) with $n=625$ and $\texttt{f}=1$. We also define a diagonal matrix $A\in\mathbb{R}^{n\times n}$ with eigenvalues $\lambda_{i}=C\frac{1}{i^{\beta}}$, exact solution $x^{\dagger}=C\cdot s_{i}\frac{1}{i^{\nu}}$ and data perturbed by noise $e_{i}=C\mathcal{N}(1,0)\frac{1}{i^{\kappa}}$, where $s_{i}\in\{-1,1\}$ are random and set the parameters as $n=20$, $\beta=4$, $\nu=2$ and $\kappa=1$.

4.1 $\ell^{1}$ regularisation

A particularly interesting application of convex variational Tikhonov regularisation is the case in which $q=1$, since it is sparsity enforcing. In fact, it is the most sparsity enforcing regularisation method whilst still remaining a convex regularisation problem. Significant work in the area of sparse regularisation includes [11, 28, 31]. Whilst it does not fit with the Muckenhoupt-type conditions we derived earlier, it is nevertheless an interesting regularisation scheme for the practitioner who would be eager to see the performance of the studied rules. Note that in this case, we minimise the Tikhonov and Bregman functionals using FISTA (cf. [5]). The corresponding proximal mapping operator is the soft thresholding operator. Note that in this experiment, we use the tomography operator defined above.

The solution $x^{\dagger}$ in our customised experiment is chosen to be sparse. For each parameter choice rule, we compute the error as

[TABLE]

We may observe in Figure (1) that, for smaller noise levels, the Hanke-Raus rule appears to be the best performing with the heuristic discrepancy rule performing similarly. The right quasi-optimality rule is particularly subpar for smaller noise levels, whilst for larger noise levels, it appears to be the best performing in fact, whereas the Hanke-Raus and heuristic discrepancy rules take a dip in performance. Indeed, we note that in various other sets of experiments we ran, instances were observed in which the heuristic discrepancy rule slightly trumped the Hanke-Raus rule.

4.2 $\ell^{\frac{3}{2}}$ regularisation

An interesting case for the purposes illustrating our theory is when $q=\frac{3}{2}$. Additionally, as with the previous regularisation, we have an analytic formula for the proximal mapping operator corresponding to the regularisation functional. In this scenario, we use the diagonal operator defined above with the given parameters and we compute the error with the Bregman distance; namely,

[TABLE]

Observe in Figure (2) that, as in the previous experiment, the Hanke-Raus rule is the best performing one in case that the noise level is relatively small, although for mid-range noise levels, the heuristic discrepancy rule performs slightly better and for larger noise levels still, the quasi-optimality rules match the heuristic discrepancy rule. Note that the quasi-optimality rules appear indistinguishable in this plot and we remark too that the plots of their respective functionals were very similar (see Figure  (3)).

The relatively poor performance of the quasi-optimality rules may be explained by a simple observation of Figure (3). In particular, one may notice that the selected minimisers of the quasi-optimality functionals are suboptimal. Note that they are selected via our procedure to choosing the interior global minimum. Indeed, if the other local minima were selected (e.g. those left of $\alpha=10^{-2}$), then the results would be much improved. This is a common phenomena in many of our experiments involving the diagonal operator with $q=\frac{3}{2}$. Indeed, we observe that the HD and HR functionals oscillate as well, although in Figure (3), at least, the correct minimisers were chosen.

4.3 $\ell^{3}$ regularisation

Based on the Muckenhoupt-type conditions in the preceding sections, we postulated that for $q>2$, the parameter choice rules we consider are likely to face mishaps. Consequently, we have elected to run a numerical experiment with $q=3$ in order to illustrate what happens in practice. As in the previous experiment, we consider the diagonal operator and compute the error induced by the parameter choice rules as before.

In Figure (4), all the rules appear to perform poorly in case the noise level is very small, but perform well overall thereafter, barring the case where the noise level is 10%, in which case the Hanke-Raus rule is the only one which produces a reasonable error relative to the optimal parameter choice. Note that in the plots of the functionals themselves, we observed that for certain noise levels, some of the functionals did not exhibit reasonable minima.

4.4 TV regularisation

Selecting

[TABLE]

with div denoting the divergence and $\Omega\subset\mathbb{R}^{n}$ an open subset, yields total variation (TV) regularisation. For the numerical treatment and for functions on the real line, this is often discretised as $\mathcal{R}=\sum\|\Delta x\|_{\ell^{1}}$ with a (e.g. forward) difference operator $\Delta$. For our numerical implementation, we used the FISTA algorithm with the proximal mapping operator for the total variation functional being computed using a fast Newton-type method, courtesy of the code provided by [2, 3].

Note that in this case, we choose $\alpha_{\text{max}}$ such that $\|x^{\delta}_{\alpha_{\text{max}}}\|\leq C$ for a reasonable constant. Moreover, for each parameter choice rule, we compute the error as

[TABLE]

which was suggested in [17]. In this instance, we consider the tomography operator.

One may observe in Figure (5) that the heuristic discrepancy rule appears to overall be the best performing one for all tested noise levels, although for larger noise levels, it is matched and/or trumped by the right quasi-optimality rule. The Hanke-Raus rule, on the other hand, does not appear to present itself as a preferable parameter choice rule in any of the tested noise levels, in this setting at least; that is, unless we compare it with the symmetric quasi-optimality rule which is the worst performing for mid-range noise levels particularly.

4.5 Summary

To summarise the numerical experiments presented above, we begin by remarking that the rules worked well, even in instances contrary to the expectations set by the theory. We observed that while none of the studied parameter choice rules were completely immune to mishaps, the heuristic discrepancy rule could perhaps be said to be the most robust overall. Indeed, in light of the above experiments, it is difficult to offer a particular recommendation.

5 Conclusion

In conclusion, we introduced four heuristic parameter choice rules for convex Tikhonov regularisation and presented a detailed analysis of the conditions we postulated for when the aforementioned rules are convergent regularisation methods. This involved the more general auto-regularisation conditions, as well as the reduction to the more specific Muckenhoupt-type conditions in case the forward operator is diagonal and the vector spaces are $\ell^{q}$. Indeed, the analysis for the heuristic discrepancy and Hanke-Raus rules was more in-depth and further investigation of the conditions presented for the symmetric quasi-optimality rule presents room for further research.

We furthermore provided a numerical study to demonstrate the performance of the rules examined in this paper and also illustrate our theoretical findings. Indeed, all the rules in question performed at least reasonably well overall and allow one to conclude that heuristic rules present themselves as viable and in fact, on occasion, attractive options for convex Tikhonov regularisation.