A new regularization approach for numerical differentiation

TL;DR

This paper introduces a novel iterative regularization method for numerical differentiation that avoids direct use of noisy data in the functional, enhancing robustness and reducing overfitting compared to traditional methods.

Contribution

The paper proposes a new iterative regularization approach that does not incorporate noisy data directly into the functional, improving stability and robustness in numerical differentiation.

Findings

The new method is robust to high noise levels and non-zero mean errors.

It avoids overfitting by ensuring the sequence converges weakly, not strongly, to noisy data.

Comparative examples show improved performance over traditional regularization techniques.

Abstract

The problem of numerical differentiation can be thought of as an inverse problem by considering it as solving a Volterra equation. It is well known that such inverse integral problems are ill-posed and one requires regularization methods to approximate the solution appropriately. The commonly practiced regularization methods are (external) parameter-based like Tikhonov regularization, which have certain inherent difficulties like choosing an optimal value of the regularization parameter, especially in the absence of noise information, which is a non-trivial task. An attractive alternative is iterative regularization, where one minimizes an associated functional in a regularized fashion (i.e., stopping at an appropriate iteration). However, in most of the regularization methods the associated minimizing functional contains the noisy data directly, and hence, the recovery gets affected. In this paper, we propose an iterative regularization method where the minimizing functional does not contain the noisy data directly. The advantage, in addition to circumventing the use of noisy data directly, is that the sequence of functions (or curves) constructed during the descent process does not converge strongly to the noise data, rather weakly and hence, avoids overfitting. Furthermore, this method is very robust to extreme noise level in the data as well as to errors with non-zero mean. To demonstrate the effectiveness of the new method we provide some examples comparing the numerical results obtained from our method with results obtained from some of the popular regularization methods such as Tikhonov regularization, total variation, smoothing spline and the polynomial regression method.

Figures

Tables

Table 1. Table 1 .

Relative errors in derivative approximation
Methods	m=100(h=0.01), $\sigma =0.01$	m=100(h=0.01), $\sigma =0.1$	m=10(h=0.1), $\sigma =0.01$
Degree-2 polynomial	0.0287	0.3190	0.2786
Tikhonov, k = 0	0.7393	0.8297	0.7062
Tikhonov, k = 1	0.1803	0.3038	0.6420
Tikhonov, k = 2	0.0186	0.0301	0.4432
Cubic Spline	0.1060	1.15	0.3004
Convolution smoothing	0.1059	0.8603	0.2098
Variational method	0.1669	0.7149	0.3419
Our method (k=0)	0.0607	0.0839	0.1355

Table 2. Table 2 .

Relative errors and the descent rates using different gradients
m=100 (h=0.01), extreme error $\sigma =0.1$	Sobolev gradient (${\nabla }_{{\mathscr{H}}^1}G$)	${ℒ}^2$-${\mathscr{H}}^1$ conjugate gradient	${\mathscr{H}}^1$-${\mathscr{H}}^1$ conjugate gradient
recovery errors (using SC-I §5.3, known $\delta$)	0.0839	0.4364	0.1522
recovery errors (using SC-II §8, unknown $\delta$)	0.1299	0.1299	0.1299
# iterations (using SC-I §5.3, known $\delta$)	39	4	6
# iterations (using SC-II §8, unknown $\delta$)	2	2	2

Table 3. Table 3 .

Relative errors and the descent rates using different gradients
m=100 (h = 0.01), moderate error $\sigma =0.01$	Sobolev gradient (${\nabla }_{{\mathscr{H}}^1}G$)	${ℒ}^2$-${\mathscr{H}}^1$ conjugate gradient	${\mathscr{H}}^1$-${\mathscr{H}}^1$ conjugate gradient
recovery errors (using SC-I §5.3, known $\delta$)	0.0607	0.0589	0.0613
recovery errors (using SC-II §8, unknown $\delta$)	0.1129	0.1129	0.1129
# iterations (using SC-I §5.3, known $\delta$)	84	5	18
# iterations (using SC-II §8, unknown $\delta$)	3	3	3

Equations

$$T_D\phi =g_1,$$

$$(T_D\phi )(x):={\int }_a^x\phi (\xi )d\xi ,$$

$$G_1(\psi )=||T_D\psi -g_1|{|}_2^2.$$

$$G_2(\psi ,\beta )=||T_D\psi -g_1|{|}_2^2+\beta ||D\psi ||,$$

$$T_D\phi =-u_1^{\prime \prime },$$

$$u_1(x)={\int }_x^b{\int }_a^{\eta }g(\xi )d\xi d\eta -g(a)\left[\frac{(b-a{)}^2}{2}-\frac{(x-a{)}^2}{2}\right].$$

$$G(\psi )=||u_{\psi }^{\prime }-u_1^{\prime }|{|}_2^2,$$

$$-u_{\psi }^{\prime \prime }$$

$$u_{\psi }(a)=u_1(a),$$

$$\tilde{g}(x)=g(x)+ϵ(x),$$

$$u_1(b)=0\text{mbox}and{u}_1(a)={\int }_a^b{\int }_a^{\eta }g(\xi )d\xi d\eta -g(a)\frac{(b-a{)}^2}{2}$$

$$u(x)$$

$$-\frac{b-x}{b-a}\left[{\int }_a^b{\int }_a^{\eta }g(\xi )d\xi d\eta -g(a)\frac{(b-a{)}^2}{2}\right],$$

$$(-\Delta v,v{)}_{{\mathcal{L}}^2}$$

$$=[-v\nabla v{]}_a^b+{\int }_a^b|\nabla v{|}^{2}dx$$

$$=(\nabla v,\nabla v{)}_{{\mathcal{L}}^2}\ge 0$$

$$G(\psi )=||u_{\psi }^{\prime }-u^{\prime }|{|}_2.$$

$$G(\psi )=||u^{\prime }-u_{\psi }^{\prime }|{|}_{{\mathcal{L}}^2}^2\le \sum _{j=0}^1||\frac{{\partial }^j}{\partial x^j}(u-u_{\psi })|{|}_{{\mathcal{L}}^2}^2=||u-u_{\psi }|{|}_{{\mathcal{H}}^1}^2$$

$$G(\psi )$$

$$\ge {\lambda }_1||u-u_{\psi }|{|}_{{\mathcal{L}}^2}^2,$$

$${\left(1+\frac{1}{{\lambda }_1}\right)}^{-1}||u-u_{\psi }|{|}_{{\mathcal{H}}^1}^2\le G(\psi )=||u^{\prime }-u_{\psi }^{\prime }|{|}_{{\mathcal{L}}^2}^2\le ||u-u_{\psi }|{|}_{{\mathcal{H}}^1}^2.$$

$$G(\psi )$$

$$={\int }_a^b({u^{\prime }}^2-{u_{\psi }^{\prime }}^2)-2(T_D\psi )(u-u_{\psi })dx.$$

$$G({\psi }_1)-G({\psi }_2)={\int }_a^b-2(T_D({\psi }_1-{\psi }_2))(u-\frac{u_{{\psi }_1}+u_{{\psi }_2}}{2})dx.$$

$$G^{\prime }(\psi )[h]={\int }_a^b(T_{D}h)(-2(u-u_{\psi }))dx,$$

$${\nabla }_{{\mathcal{L}}^2}^{\psi }G=T_D^{\ast }(-2(u-u_{\psi })),$$

$$(T_D^{\ast }f)(x)={\int }_x^{b}f(\xi )d\xi ;\forall x\in [a,b].$$

$$G^{\prime \prime }(\psi )[h,k]=2(-{\Delta }^{-1}(T_{D}h),(T_{D}k){)}_{{\mathcal{L}}^2}$$

$$\underset{ϵ\to 0}{\lim }{u}_{\psi +ϵh}=u_{\psi }.$$

$$-u_{\psi }^{\prime \prime }$$

$$-u_{\psi +ϵh}^{\prime \prime }$$

$$-\Delta (u_{\psi +ϵh}-u_{\psi })=ϵT_{D}h$$

$$(\nabla (u_{\psi +ϵh}-u_{\psi }),\nabla (u_{\psi +ϵh}-u_{\psi }){)}_{{\mathcal{L}}^2}=ϵ(T_{D}h,u_{\psi +ϵh}-u_{\psi }{)}_{{\mathcal{L}}^2}.$$

$$(1+{\lambda }_1^{-1}{)}^{-1}||u_{\psi +ϵh}-u_{\psi }|{|}_{{\mathcal{H}}^1}^2\le ||\nabla (u_{\psi +ϵh}-u_{\psi })|{|}_{{\mathcal{L}}^2}^2=ϵ(T_{D}h,u_{\psi +ϵh}-u_{\psi }{)}_{{\mathcal{L}}^2},$$

Full text

1. Introduction

In many applications we want to calculate the derivative of a function measured experimentally, that is, to differentiate a function obtained from discrete noisy data. The problem consists of calculating stably the derivative of a smooth function $g$ given its noisy data $\tilde{g}$ such that $\lvert\lvert\tilde{g}-g\rvert\rvert\leq\delta$, where the norm $\lvert\lvert.\rvert\rvert$ can be either the $\mathcal{L}^{\infty}\text{ or }{\mathcal{L}^{2}}$-norm. This turns out to be an ill-posed problem, since for a small $\delta$ such that $\lvert\lvert\tilde{g}-g\rvert\rvert\leq\delta$ we can have $\lvert\lvert\tilde{g}^{\prime}-g^{\prime}\rvert\rvert$ arbitrarily large or even throw $\tilde{g}$ out of the set of differentiable functions. Many methods and techniques have been introduced in the literature regarding this topic (see [4], [8], [9], [10], [11], [12], [14], [15], [16], [26], [27], [28], [29] and references therein). They mostly fall into one of three categories: difference methods, interpolation methods and regularization methods. One can use the first two methods to get satisfactory results provided that the function is given precisely, but they fail miserably when they encounter even small amounts of noise, if the step size is not chosen appropriately or in a regularized way. Hence in such scenarios regularization methods needs to be employed to handle the instability arising from noisy data, with Tikhonov’s regularization method being very popular in this respect. In a regularization method the instability is bypassed by reducing the numerical differentiation problem to a family of well-posed problems, depending on a regularization (smoothing) parameter. Once an optimal value for this parameter is found, the corresponding well-posed problem is then solved to obtain an estimate for the derivative. Unfortunately, the computation of the optimal value for this parameter is a nontrivial task, and usually it demands some prior knowledge of the errors involved. Though there are many techniques developed to find an appropriate parameter value, such as the Morzov’s discrepancy principle, the L-curve method and others, still sometimes the solution recovered is either over-fitted or over-smoothed; especially when $g$ has sharp edges, discontinuities or the data has extreme noise in it. An attractive alternative is the iterative regularization methods, like the Landweber iterations. In an iterative regularization method (without any explicit dependence on external parameters) one can start the minimization process first and then use the iteration index as an regularization parameter, i.e., one terminates the minimization process at an appropriate time to achieve regularization. More recently iterative methods have been investigated in the frame work of regularization of nonlinear problems, see [19, 23, 1, 13, 17].

In this paper we propose a new smoothing or regularization technique that doesn’t involve any external parameters, thus avoiding all the difficulties associated with them. Furthermore, since this method does not involve the noisy data directly, it is very robust to large noise level in the data as well as to errors with non-zero mean. Thus, it makes the method computationally feasible when dealing with real-life data, where we do not expect to have any knowledge of the errors involved or when there is extreme noise involved in the data.

Let us briefly outline our method. First, we lay out a brief summary of a typical (external) parameter-dependent regularization method, and then we present our new (external) parameter-independent smoothing technique. For a given differentiable function $g$ the problem of numerical differentiation, finding $\varphi=g^{\prime}$, can be expressed via a Volterra integral equation111the technicality as to how far $g$ can be weakened so that the solution of equation (1.1) is uniquely and stably recovered (finding the $\varphi$ inversely) will be discussed later.:

[TABLE]

where $g_{1}(x)=g(x)-g(a)$ and

[TABLE]

for all $x\in[a,b]\subset\mathbb{R}$. Here one can attempt to solve equation (1.1) by approximating $\varphi$ with the minimizer of the functional

[TABLE]

However, as stated earlier, the recovery becomes unstable for a noisy $g$. So, to counter the ill-posed nature of this problem regularization techniques are introduced: one instead minimizes the functional222again the domain space for the functionals $G_{1}$, $G_{2}$ and $G$ are discussed in details later.

[TABLE]

where $\beta$ is a regularization parameter and $D$ is a differentiation operator of some order (see [1, 4]). This converts the ill-posed problem to a family of conditionally well-posed problems depending on $\beta$. The first term (fitting term) of the functional $G_{2}$ ensures that the inverse solution fits well with the given data, when applied by the forward operator $T_{D}$, and the second term (smoothing term) in (1.3) controls the smoothness of the inverse solution. Since the minimzer of the second term is not the solution of the inverse problem, unless it is a trivial solution, one needs to balance between the smoothing and fitting of the inverse recovery by finding an optimal value $\beta_{0}$. Then the exact solution $\varphi$ is approximated by minimizing the corresponding functional $G(.,\beta_{0})$.

Given that an important key in any regularization technique is the conversion of an ill-posed problem to a (conditionally) well-posed problem, we make this our main goal. We start with integrating the data twice, which helps in smoothing out the noise. Thus (1.1) can be reformulated as: find a $\varphi$ that satisfies

[TABLE]

where $-u_{1}^{\prime\prime}=g_{1}$ and $g_{1}^{\prime}=g^{\prime}=\varphi$. Equivalently, $u_{1}(x)=-\left(-\int_{x}^{b}\int_{a}^{\eta}\;g_{1}(\xi)\;d\xi d\eta\right)$ or

[TABLE]

Now we find the solution of (1.4) by approximating it with the minimzer of the following functional:

[TABLE]

where $u_{\psi}$ is the solution, for a given $\psi$, of the following boundary value problem:

[TABLE]

Note that unlike the functionals $G_{1}$ and $G_{2}$ which are defined on the data $g$ directly, the functional $G$ is defined rather on the transformed data $u_{1}^{\prime}$. Thus, one can see that when the data $g$ has noise in it then working with the smoothed (integrated) data $u_{1}$ is more effective than working with the noisy data $g$ directly. It will be proved in later sections that this simple change in the working space from $g$ to $u_{1}$ drastically improves the stability and smoothness of the inverse recovery of $\varphi$, without adding any further smoothing terms.

Remark 1.1*.*

Typically, real-life data have zero-mean additive noise in it, i.e.,

[TABLE]

for $x\in[a,b]$, such that $\mathbb{E}_{\mu}(\epsilon)=0$ and $\mathbb{E}_{\mu}(\epsilon^{2})\leq\delta$ (or equivalently, $\lvert\lvert\tilde{g}-g\rvert\rvert_{\mathcal{L}^{2}}\leq\delta$), where $\mu$ is the probability density function of the random variable $\epsilon$. Therefore, integrating the data smooths out the noise present in the original data ($\tilde{g}$) and hence, the new data ($\tilde{u_{1}}$, as defined by (1.5) for $\tilde{g}$) for the transformed equation (1.4) has a significantly reduced noise level in it, see the comparison in Figure 5.

We prove, in Section 3, that the functional $G$ is strictly convex and hence has a unique global minimum which satisfies (1.1), that is, $\varphi=\operatorname*{arg\,min}_{\psi}\;G(\psi)$ satisfies $g^{\prime}=\varphi$. The global minimum is achieved via an iterative method using an upgraded steepest gradient or conjugate-gradient method, presented in Section 4, where the use of an (Sobolev) $\mathcal{H}^{1}$-gradient for $G$ instead of the commonly used $\mathcal{L}^{2}$-gradient is discussed and is a crucial step in the optimization process. In Section 5 we shed some light on the stability, convergence and well-posedness of this technique. In Section 6 the Volterra operator $T_{D}$ is improved for better computation and numerical efficiency but keeping it consistent with the theory developed. In Section 7 we provide some numerical results and compare our method of numerical differentiation with some popular regularization methods, namely Tikhonov regularization, total variation, smoothing spline, mollification method and least square polynomial approximation (the comparison is done with results provided in [4], [2] and [8]). Finally, in Section 8 we present an efficient (heuristic) stopping criterion for the descent process when we don’t have any prior knowledge of the error norm.

2. Notations and Preliminaries

We adopt the following notations that will be used throughout the paper. All functions are real-valued defined on a bounded closed domain $[a,b]\subset\mathbb{R}$. For $1\leq p<\infty$, $\mathcal{L}^{p}[a,b]$ := $(\mathcal{L}^{p},\lvert\lvert.\rvert\rvert_{\mathcal{L}^{p}},[a,b])$ denotes the usual Banach space of $p$-integrable functions on $[a,b]$ and the space $\mathcal{L}^{\infty}[a,b]:=$ $(\mathcal{L}^{\infty},||.||_{\mathcal{L}^{\infty}},[a,b])$ contains the essentially bounded measurable functions. Likewise the Sobolev space $\mathcal{H}^{q}[a,b]:=$ $(\mathcal{H}^{q},||.||_{\mathcal{H}^{q}},[a,b])$ contains all the functions for which $f$, $f^{\prime}$, $\cdots,$ $f^{(q)}\;\in$ $\mathcal{L}^{2}[a,b]$, with weak differentiation understood, and the space $\mathcal{H}^{q}_{0}[a,b]:=\{f\in\mathcal{H}^{q}:``f\mbox{ vanishes at the boundary"}\}$. The spaces $\mathcal{L}^{2}[a,b]$ and $\mathcal{H}^{q}$ are Hilbert spaces with inner-products denoted by $(.,.)_{\mathcal{L}^{2}}$ and $(.,.)_{\mathcal{H}^{q}}$, respectively.

Remark 2.1*.*

Note that although the Volterra operator $T_{D}$ is well-defined on the space of integrable functions ($\mathcal{L}^{1}[a,b]$), we will restrict it to the space of $\mathcal{L}^{2}[a,b]$ functions, that is, we shall consider $\mathcal{L}^{2}[a,b]$ to be the searching space for the solution of (1.1), since it is a Hilbert space and has a nice inner product $(.,.)_{\mathcal{L}^{2}}$. Hence the domain of the functional $G$ is $\mathcal{D}_{G}=\mathcal{L}^{2}[a,b]$. It is not hard to see that the Volterra operator $T_{D}$ is linear and bounded on $\mathcal{L}^{2}[a,b]$.

Remark 2.2*.*

From Remark 2.1, since $\varphi\in\mathcal{L}^{2}[a,b]$ we have $T_{D}\varphi\in\mathcal{L}^{2}[a,b]$. Thus our problem space to be considered is $\mathcal{H}^{1}[a,b]$, that is, $g\in\mathcal{H}^{1}[a,b]$. Note that since we will not be working with $g$ directly but rather with $u_{1}$, integrating twice makes $u_{1}\in\mathcal{H}^{3}[a,b]\subset\mathcal{H}^{1}[a,b]$. This is particularly significant in the sense that we are able to upgrade the smoothness of the working data or information space from $\mathcal{H}^{1}[a,b]$ to $\mathcal{H}^{3}[a,b]$ and hence improve the stability and efficiency of numerical computations. This effect can be seen in some of the numerical examples presented in Section 7 where the noise involved in the data is extreme.

Before we proceed to prove the convexity of $G$ and the well-posedness of the method, we will manipulate $u_{1}$ further to push it to an even nicer space. From equation (1.5), we have

[TABLE]

Now redefining the working data as $u(x)=u_{1}(x)-\frac{b-x}{b-a}u_{1}(a)$, we have

[TABLE]

and this gives $u(a)=0$, $u(b)=0$ and $-u^{\prime\prime}=-u_{1}^{\prime\prime}=g_{1}$. Thus $u\in\mathcal{H}^{3}_{0}[a,b]$ and the negative Laplace operator $-\Delta\;=-\frac{\partial^{2}}{\partial x^{2}}$ is a positive operator in $\mathcal{L}^{2}[a,b]$ on the domain $\mathcal{D}_{\Delta}=\mathcal{H}^{2}_{0}[a,b]\supset\mathcal{H}^{3}_{0}[a,b]$, since for any $v\in\mathcal{H}^{2}_{0}[a,b]$

[TABLE]

and equality holds when $\nabla v=0$, which implies $v\equiv 0$. Here, and until otherwise specified, the notations $\Delta$ and $\nabla$, will mean $\frac{\partial^{2}}{\partial x^{2}}$, $\frac{\partial}{\partial x}$, respectively, throughout the paper. The functional $G(\psi)$ in (1.6) is now redefined, with the new $u$, as

[TABLE]

The positivity of the operator $-\Delta$ in $\mathcal{L}^{2}[a,b]$ on the domain $\mathcal{D}_{\Delta}$ also helps us to obtain an upper and lower bound for $G(\psi)$, for any $\psi\in\mathcal{L}^{2}[a,b]$. First we get the trivial upper bound

[TABLE]

To get a lower bound we note from (1.8) that $v=u-u_{\psi}\in\mathcal{H}^{2}_{0}[a,b]$, thus

[TABLE]

where $\lambda_{1}>0$ is the smallest eigenvalue of the positive operator $-\Delta$ on $\mathcal{D}_{\Delta}$. Hence, we get the bounds of $G(\psi)$ in terms of the $\mathcal{H}^{1}[a,b]$ norm of $u-u_{\psi}$ as

[TABLE]

Equation 2.6 indicates the stability of the method and the convergence of $u_{\psi_{m}}$ to $u$ in $\mathcal{H}^{1}[a,b]$ if $G(\psi_{m})\rightarrow 0$, as is explained in detail in Section 5.

3. Convexity of the functional $G$

In this section we prove the convexity of the functional $G$, together with some important properties.

Theorem 3.1**.**

---

• (i)

An equivalent form of $G$, for any $\psi\in\mathcal{L}^{2}[a,b]$, is :

[TABLE]

• (ii)

For any $\psi_{1},\;\psi_{2}\in\mathcal{L}^{2}[a,b]$, we have

[TABLE]

• (iii)

The first G$\hat{a}$teaux derivative333it can be further proved that it’s also the first Fr$\acute{e}$chet derivative of G at $\psi$., at $\psi\in\mathcal{L}^{2}[a,b]$, for G is given by

[TABLE]

where $h\in\mathcal{L}^{2}[a,b]$. And the $\mathcal{L}^{2}$-gradient of $G$, at $\psi$, is given by

[TABLE]

where the adjoint $T_{D}^{*}$ of $T_{D}$ is given by, for any $f\in\mathcal{L}^{2}[a,b]$,444Hence $T_{D}^{*}$ is also a linear and bounded operator in $\mathcal{L}^{2}[a,b]$.

[TABLE]

• (iv)

The second G$\hat{a}$teaux derivative 555again, it can be proved that it’s the second Fr$\acute{e}$chet derivative of G at $\psi$., at any $\psi\in\mathcal{L}^{2}[a,b]$, of G is given by

[TABLE]

where $h,\;k\in\mathcal{L}^{2}[a,b]$. Hence for any $\psi\in\mathcal{L}^{2}[a,b]$, $G^{\prime\prime}(\psi)$ is a positive definite quadratic form.

For the proof of Theorem 3.1 we also need the following ancillary result.

Lemma 3.2**.**

For fixed $\psi,\;h\in\mathcal{L}^{2}[a,b]$ we have, in $\mathcal{H}^{1}[a,b]$,

[TABLE]

Proof.

Subtracting the following equations

[TABLE]

we have

[TABLE]

and using $u_{\psi+\epsilon h}-u_{\psi}\in\mathcal{H}^{2}_{0}[a,b]$ we get, via integration by parts,

[TABLE]

Now from (2) we have ${\lambda_{1}}\lvert\lvert u_{\psi+\epsilon h}-u_{\psi}\rvert\rvert_{\mathcal{L}^{2}}^{2}\leq\lvert\lvert\nabla(u_{\psi+\epsilon h}-u_{\psi})\rvert\rvert_{\mathcal{L}^{2}}^{2}$, and using the Cauchy-Schwarz inequality, we have

[TABLE]

where $\lambda_{1}>0$. Hence $u_{\psi+\epsilon h}\xrightarrow{\epsilon\rightarrow 0}u_{\psi}$ in $\mathcal{H}^{1}[a,b]$, since the operator $T_{D}$ is bounded and $\psi,\;h\in\mathcal{L}^{2}[a,b]$ are fixed, which implies the right hand side is of $O(\epsilon)$. ∎

3.1. Proof of Theorem 3.1

The proof of first two properties (i) and (ii) are straight forward via integration by parts and using the fact that $u-u_{\psi}\in\mathcal{H}^{2}_{0}[a,b]$. In order to prove (iii) and (iv), we use Lemma 3.2.

• (iii)

The Gâteaux derivative of the functional $G$ at $\psi$ in the direction of $h\in\mathcal{L}^{2}[a,b]$ is given by

[TABLE]

Now for a fixed $\epsilon>0$, we have using 3.8,

[TABLE]

Using Lemma 3.2, one obtains the Gâteaux derivative of $G$ at $\psi\in\mathcal{L}^{2}[a,b]$ in the direction of $h\in\mathcal{L}^{2}[a,b]$ as

[TABLE]

Note that $(T_{D}h,-2(u-u_{\psi}))_{\mathcal{L}^{2}}=(h,T_{D}^{*}(-2(u-u_{\psi})))_{\mathcal{L}^{2}}$ for all $h\in\mathcal{L}^{2}[a,b]$, where $T_{D}^{*}$ is the adjoint of the operator $T_{D}$. Hence by Riesz representation theorem, the $\mathcal{L}^{2}$-gradient of the functional $G$ at $\psi$ is given by

[TABLE]

• (iv)

Finally, the second Gâteaux derivative for the functional $G$ at $\psi$ is given by

[TABLE]

Again for a fixed $\epsilon>0$, we have using (3.8)

[TABLE]

Hence from (3.10) and letting $\epsilon\rightarrow 0$ we get

[TABLE]

Here we can see the strict convexity of the functional $G$, as for any $h\in\mathcal{L}^{2}[a,b]$, we have

[TABLE]

where $-\Delta y=T_{D}h$ and $y\in\mathcal{H}^{2}_{0}[a,b]$ (from (3.8)). As $-\Delta$ is a positive operator on $\mathcal{H}^{2}_{0}[a,b]$, $y$ is the trivial solution if and only if $T_{D}h=0$. But

[TABLE]

for all $x\in[a,b]$ if and only if $h\equiv 0$. Thus $G^{\prime\prime}(\psi)$ is a positive definite form for any $\psi\in\mathcal{L}^{2}[a,b]$. ∎

In this section we proved that the functional $G$ is strictly convex and hence has a unique minimizer, which is attained by the solution $\varphi$ of the inverse problem (1.1). We next discuss a descent algorithm that uses the $\mathcal{L}^{2}$-gradient to derive other gradients that provide descent directions, with better and faster descent rates.

4. The Descent Algorithm

Here we discuss the problem of minimizing the functional $G$ via a descent method. Theorem 3.1 suggests that the minimization of the functional $G$ should be computationally effective in that $\varphi$ is not only the unique global minimum for $G$ but also the unique zero for the gradient, that is, $\nabla_{\mathcal{L}^{2}}^{\psi}G\neq 0$ for $\psi\neq\varphi$. Now for a given $\psi\in\mathcal{L}^{2}[a,b]$, let $h\in\mathcal{L}^{2}[a,b]$ denote an update direction for $\psi$. Then Taylor’s expansion gives

[TABLE]

or, for sufficiently small $\alpha>0$, we have

[TABLE]

So if we choose the direction $h$ in such a way that $G^{\prime}(\psi)[h]>0$, then we can minimize $G$ along this direction. Thus we can set up a recovery algorithm for $\varphi$, forming a sequence of values

[TABLE]

We list a number of different gradient directions that can make $G^{\prime}(\psi)[h]>0$.

1. (1)

The $\mathcal{L}^{2}$-Gradient:

First, notice from Theorem 3.1 that at a given $\psi\in\mathcal{L}^{2}[a,b]$,

[TABLE]

so if we choose the direction $h=\nabla_{\mathcal{L}^{2}}^{\psi}G$ at $\psi$, then $G^{\prime}(\psi)[h]>0$. However, there are numerical issues associated with $\mathcal{L}^{2}$-gradient of $G$ during the descent process stemming from the fact that it is always zero at $b$. Consequently, the boundary data at $b$ for the evolving functions $\psi_{m}$ are invariant during the descent and there is no control on the evolving boundary data at $b$. This can result in severe decay near the boundary point $b$ if $\psi_{initial}(b)\neq\varphi(b)$, as the end point for all such $\psi_{m}$ is glued to $\psi_{initial}(b)$, but $\psi_{m}\rightarrow\varphi$ in $\mathcal{L}^{2}[a,b]$, as is proved in section 5. 2. (2)

The $\mathcal{H}^{1}$-Gradient:

One can circumvent this problem by opting for the Sobolev gradient $\nabla_{\mathcal{H}^{1}}^{\psi}G$ instead (see [6]), which is also known as the Neuberger gradient. It is defined as follows: for any $h\in\mathcal{H}^{1}[a,b]$

[TABLE]

where $g=\nabla_{\mathcal{H}^{1}}^{\psi}G$. Comparing with (4.2) one can obtain the Neuberger gradient $g$ at $\psi$, by solving the boundary value problem

[TABLE]

Setting $h=g$, the boundary condition becomes

[TABLE]

This provides us a gradient, $\nabla_{\mathcal{H}^{1}}^{\psi}G$, with considerably more flexibility at the boundary points $a$ and $b$. In particular consider the following cases:

1. (a)

Dirichlet Neuberger gradient : $g(a)=0$ and $g(b)=0$. 2. (b)

Neumann Neuberger gradient : $g^{\prime}(a)=0$ and $g^{\prime}(b)=0$. 3. (c)

Robin or mixed Neuberger gradient : $g(a)=0$ and $g^{\prime}(b)=0$ or $g^{\prime}(a)=0$ and $g(b)=0$.

This excellent smoothing technique was originally introduced and used by Neuberger. In addition to the flexibility at the end points, it enables the new gradient to be a preconditioned (smoothed) version of $\nabla_{\mathcal{L}^{2}}^{\psi}G$, as $g=(I-\Delta)^{-1}\;\nabla_{\mathcal{L}^{2}}^{\psi}G$, and hence gives a superior convergence in the steepest descent algorithms. So now choosing the descent direction $h=g$ at $\psi$ makes $G^{\prime}(\psi)[h]=(g,\nabla_{\mathcal{H}^{1}}^{\psi}G)_{\mathcal{H}^{1}}>0$ and hence $G(\psi-\alpha h)-G(\psi)<0$ (from (4.1)). As stated earlier, the greatest advantage of this gradient is the control of boundary data during the descent process, since based on some prior information of the boundary data we can choose any one of the three aforementioned gradients. For example, if some prior knowledge on $\varphi(a)$ and $\varphi(b)$ are known, then one can define $\varphi_{initial}$ as the straight line joining them and use the Dirichlet Neubeger gradient for the descent. Thus the boundary data is preserved in each of the evolving $\psi_{m}$ during the descent process, which leads to a much more efficient, and faster, descent compared to the normal $\mathcal{L}^{2}$-gradient descent. Even when $\varphi|_{\{a,b\}}$ is unknown, one can use the Neumann Neuberger gradient that allows free movements at the boundary points rather than gluing it to a fixed value. In the latter scenario, one can even take the average of $\nabla_{\mathcal{H}^{1}}^{\psi}G$ and $\nabla_{\mathcal{L}^{2}}^{\psi}G$ to make use of both the gradients. 3. (3)

The $\mathcal{L}^{2}-\mathcal{H}^{1}$ Conjugate Gradient:

If one wishes to further boost the descent speed (by, roughly, a factor of two) and make the best use of both the gradients, then the standard Polak-Ribi$\grave{e}$re conjugate gradient scheme (see [7]) can be implemented. The initial search direction at $\psi_{0}$, is $h_{0}=g_{0}=\nabla_{\mathcal{H}^{1}}^{\psi_{0}}G$. At $\psi_{m}$ one can use the exact or inexact line search routine to minimize $G(\psi)$ in the direction of $h_{m}$ resulting in $\psi_{m+1}$. Then $g_{m+1}=\nabla_{\mathcal{H}^{1}}^{\psi_{m+1}}G$ and $h_{m+1}=g_{m+1}+\gamma_{m}h_{m}$, where

[TABLE]

Remark 4.1*.*

Though the $\mathcal{L}^{2}$-$\mathcal{H}^{1}$ conjugate gradient boost the descent rate, it (sometimes) compromises the accuracy of the recovery, especially when the noise present in the data is extreme, see Table 2 when $\sigma=0.1$. Now one can also construct a $\mathcal{H}^{1}$-$\mathcal{H}^{1}$ conjugate gradient based only on the Sobolev gradient. This is smoother than the $\mathcal{L}^{2}$-$\mathcal{H}^{1}$ conjugate gradient (as Sobolev gradients are smoother, see equation (4.3)) and hence improves the accuracy of the recovered solution (specially for smooth recovery), but it is tad slower than the $\mathcal{L}^{2}$-$\mathcal{H}^{1}$ conjugate gradient. Finally, the simple Sobolev gradient ($\nabla_{\mathcal{H}^{1}}G$) provides the most accurate recovery, but at the cost of the descent rate (it the slowest amongst the three), see Tables 2 and 3.

4.1. The Line Search Method

We minimize the single variable function $f_{m}(\alpha)=G(\psi_{m+1}(\alpha))$, where $\psi_{m+1}(\alpha)=\psi_{m}-\alpha\nabla_{\mathcal{H}^{1}}^{\psi_{m}}G$, via a line search minimization by first bracketing the minimum and then using some well-known optimization techniques like Brent minimization to further approximate it. Note that the function $f_{m}(\alpha)$ is strictly decreasing in some neighborhood of $\alpha=0$ as $f_{m}^{\prime}(0)=-\lvert\lvert g_{m}\rvert\rvert_{\mathcal{H}^{1}}^{2}<0$.

In order to achieve numerical efficiency, we need to carefully choose the initial step size $\alpha_{0}$. For that, we use the quadratic approximation of the function $f_{m}(\alpha)$ as follows

[TABLE]

which gives the minimizing value for $\alpha$ as

[TABLE]

Now since $\alpha_{0}$ is derived from the quadratic approximation of the functional $G$, it is usually very close to the optimal value, thereby reducing the computational time of the descent algorithm significantly. Now if, for $k=0,1,2\cdots$, $f_{m}((k+1)\alpha_{0})>f_{m}(k\alpha_{0})$ then we have a bracket, $[\max\{k-1,0\}\alpha_{0},(k+1)\alpha_{0}]$, for the minimum and one can use single variable minimization solvers to approximate it.

In this section we saw a descent algorithm, with various gradients, where starting from an initial guess $\psi_{0}\in\mathcal{L}^{2}[a,b]$, we obtain a sequence of $\mathcal{L}^{2}$-functions $\psi_{m}$ for which the sequence $\{G(\psi_{m})\}$ is strictly decreasing. In the next section, we discuss the convergence of the $\psi_{m}$’s to $\varphi$ and the stability of the recovery.

5. Convergence, Stability and Conditional Well-posedness

Exact data: We first prove that the sequence of functions constructed during the descent process converges to the exact source function in the absence of any error term and then proves the stability of the process in the presence of noise in the data.

5.1. Convergence

First we see that for the sequence $\{\psi_{m}\}$ produced by the steepest descent algorithm, described in Section 4, we have $G(\psi_{m})\rightarrow 0$, since the functional $G$ is non-negative and strictly convex (with the global minimizer $\varphi$, $G(\varphi)=0$) and $G(\psi_{m+1})<G(\psi_{m})$. In this subsection we prove that if for any sequence of functions $\{\psi_{m}\}\subset\mathcal{L}^{2}[a,b]$ such that $G(\psi_{m})\rightarrow 0$ then $\psi_{m}\xrightarrow{w}\varphi$ in $\mathcal{L}^{2}[a,b]$, where $\xrightarrow{w}$ denotes weak convergence in $\mathcal{L}^{2}[a,b]$.

Theorem 5.1**.**

Suppose that $\{\psi_{m}\}$ is any sequence of $\mathcal{L}^{2}$-functions such that the sequence $\{G(\psi_{m})\}$ tends to zero. Then $\{\psi_{m}\}$ converges weakly to $\varphi$ in $\mathcal{L}^{2}[a,b]$ and $\{u_{\psi_{m}}\}$ converges strongly to $u$ in $\mathcal{H}^{1}[a,b]$. Also, the sequence $\{g_{m}\}$ converges weakly to $g_{1}$ in $\mathcal{L}^{2}[a,b]$, where $g_{m}=-u_{\psi_{m}}^{\prime\prime}$ and $g_{1}=-u^{\prime\prime}$.

Proof.

The proof of $u_{\psi_{m}}\xrightarrow{s}u$ in $\mathcal{H}^{1}[a,b]$ is trivial from the bounds of $G(\psi)$ in equation (2.6), which gives

[TABLE]

To see the weak convergence of $\{\psi_{m}\}$ to $\varphi$ in $\mathcal{L}^{2}[a,b]$, we first prove that the sequence $\{T_{D}\psi_{m}\}$ converges weakly to $T_{D}\varphi$ in $\mathcal{L}^{2}[a,b]$. Since $\{u_{\psi_{m}}\}$ converges strongly to $u$ in $\mathcal{H}^{1}[a,b]$, this implies $\{u_{\psi_{m}}^{\prime}\}$ and $\{u_{\psi_{m}}\}$ converge weakly to $u^{\prime}$ and $u$ in $\mathcal{L}^{2}[a,b]$, respectively. Now we will use the fact that $\mathcal{C}^{\infty}_{0}[a,b]$ is dense in $\mathcal{L}^{2}[a,b]$ (in $\mathcal{L}^{2}$-norm), i.e., for any $\psi\in\mathcal{L}^{2}[a,b]$ and $\epsilon>0$ there exists a $\tilde{\psi}\in\mathcal{C}^{\infty}_{0}[a,b]$ such that $\lvert\lvert\psi-\tilde{\psi}\rvert\rvert_{\mathcal{L}^{2}}\leq\epsilon$. So for any $\tilde{\psi}\in\mathcal{C}^{\infty}_{0}[a,b]$ we have, using $\tilde{\psi}^{\prime}\in\mathcal{L}^{2}[a,b]$,

[TABLE]

which tends to zero as $m\rightarrow\infty$. Hence by the density of $\mathcal{C}^{\infty}_{0}[a,b]$ in $\mathcal{L}^{2}[a,b]$, it can be proved that the sequence $\{T_{D}\psi_{m}\}$ converges weakly to $T_{D}\varphi$ in $\mathcal{L}^{2}[a,b]$.

To prove convergence of $\{\psi_{m}\}$ to $\{\varphi\}$ weakly in $\mathcal{L}^{2}[a,b]$, we can use $(T_{D}(\psi_{m}-\varphi),\psi)_{\mathcal{L}^{2}}=(\psi_{m}-\varphi,T_{D}^{*}\;\psi)_{\mathcal{L}^{2}}$ and 5.2. Therefore, our proof will be complete if we can show that the range of $T_{D}^{*}$ is dense in $\mathcal{L}^{2}[a,b]$ in $\mathcal{L}^{2}$-norm. Again we start with any $\tilde{\psi}\in\mathcal{C}^{\infty}_{0}[a,b]$, and using $\tilde{\psi}^{\prime}\in\mathcal{L}^{2}[a,b]$, we have

[TABLE]

i.e., $\mathcal{C}^{\infty}_{0}[a,b]\subset Range(T_{D}^{*})$. Hence we have $Range(T_{D}^{*})$ dense in $\mathcal{L}^{2}[a,b]$ in $\mathcal{L}^{2}$-norm. For convergence of $\{g_{m}\}$ to $g_{1}$, note that $T_{D}\psi_{m}=-u_{\psi_{m}}^{\prime\prime}=g_{m}$ and $T_{D}\varphi=-u^{\prime\prime}=g_{1}$. And since $\{T_{D}\psi_{m}\}$ converges weakly to $T_{D}\varphi$ in $\mathcal{L}^{2}[a,b]$ implies $\{g_{m}\}$ converges weakly to $g_{1}$ in $\mathcal{L}^{2}[a,b]$. ∎

Remark 5.2*.*

In can be further proved that the sequence $\{\psi_{m}\}$ converges strongly to $\varphi$ in $\mathcal{L}^{2}[a,b]$. To prove this, first, one needs to analyze the operator associated with the minimizing functional $G$ as defined in (2.3), i.e., from the definition of $u_{\psi}$ in (1.7) together with the criterion $u_{\psi}\in\mathcal{H}^{2}_{0}[a,b]$ we have (from (2.1), with $g=T_{D}\psi$ and $(T_{D}\psi)(a)=0$, by the definition of $T_{D}$)

[TABLE]

From the expression (5.3) the operator $L(\psi):=u_{\psi}^{\prime}$ is both linear and bounded, and hence minimizing the functional $G$ in (2.3) is equivalent to Landweber iterations corresponding to the operator $L$. Then from the convergence theories developed for Landweber iterations the sequence $\{\psi_{m}\}$ converges to $\varphi$ strongly in $\mathcal{L}^{2}[a,b]$, for details on iterative regularization see [1, 17].

Theorem 5.1 proves that for the given function $g_{1}\in\mathcal{L}^{2}[a,b]$ we are able to construct a sequence of smooth functions, $\{g_{m}\}\subset\mathcal{L}^{2}[a,b]$, that converges (weakly) to $g_{1}$ in $\mathcal{L}^{2}[a,b]$. This is critical since, when the data has noise in it one needs to terminate the descent process at an appropriate instance to attain regularization, see §5.3, and the (weak) convergence helps us to construct such a stopping criterion in the absence of noise information ($\delta$), see §8.

Noisy data: In this subsection we consider the data has noise in it and shows that the sequence of functions constructed during the descent process using the noisy data still approximates the exact solution, under some conditions.

5.2. Stability

Here we prove the stability of the process. We will prove this by considering the problem of numerical differentiation as equivalent to finding a unique minimizer of the positive functional $G$, this makes the problem well-posed. That is, for a given $g\in\mathcal{H}^{1}[a,b]$, and hence a given $u\in\mathcal{H}^{3}_{0}[a,b]$ and the functional $G$, the problem of finding (derivative) $\varphi\in\mathcal{L}^{2}[a,b]$ such that $g^{\prime}=\varphi$ is equivalent to finding the minimizer of the functional $G$, i.e., a $\psi\in\mathcal{L}^{2}[a,b]$ such that $G(\psi)\leq\epsilon$, for any small $\epsilon>0$, is a conditionally well-posed problem. It is not hard to prove that if two functions $g$, $\tilde{g}\in\mathcal{H}^{1}[a,b]$ are such that $\lvert\lvert g-\tilde{g}\rvert\rvert_{\mathcal{L}^{2}}\leq\delta$, where $\delta>0$ is small, then the corresponding $u$, $\tilde{u}\in\mathcal{H}^{3}_{0}[a,b]$ also satisfy666just for simplicity we assume $g(a)=\tilde{g}(a)$. $\lvert\lvert u-\tilde{u}\rvert\rvert_{\mathcal{H}^{1}}\leq C\delta$, for some constant $C$.

Theorem 5.3**.**

*Suppose the function $\tilde{u}$ in the perturbed version of the function $u$ such that $\lvert\lvert u-\tilde{u}\rvert\rvert_{\mathcal{H}^{1}}\leq\delta$, where $\delta>0$, and let $\varphi$, $\tilde{\varphi}\in\mathcal{L}^{2}[a,b]$ denote their respective recovered functions, such that $T_{D}\varphi=-u^{\prime\prime}$ and $T_{D}\tilde{\varphi}=-\tilde{u}^{\prime\prime}$. Let the functional $G$, without loss of generality, be defined based on $\tilde{u}$, that is, $G(\tilde{\varphi})=0$, then we have *

[TABLE]

where $C$ is some constant.

Proof.

Since $u_{\varphi}=u$ and $\tilde{u}_{\tilde{\varphi}}=\tilde{u}$, the proof follows from the definitions of the corresponding functionals, as

[TABLE]

∎

In the next theorem we prove that if a sequence of functions $\{\psi_{m}\}$ converges to $\tilde{\varphi}$ in $\mathcal{L}^{2}[a,b]$ then it also approximates $\varphi$ in $\mathcal{L}^{2}[a,b]$, that is, if $G_{\tilde{u}}(\psi_{m})$ is small then $G_{u}(\psi_{m})$ is also small where $G_{u}$ and $G_{\tilde{u}}$ are the functionals formed based on $u$ and $\tilde{u}$, respectively.

Theorem 5.4**.**

Suppose for a sequence of functions $\{\psi_{m}\}\subset\mathcal{L}^{2}[a,b]$, the corresponding sequence $\{G_{\tilde{u}}(\psi_{m})\}\subset\mathbb{R}$ converges to zero, where the functional $G_{\tilde{u}}$ is formed based on $\tilde{u}$, then for the original $u$ such that $\lvert\lvert u-\tilde{u}\rvert\rvert_{\mathcal{H}^{1}}\leq\delta$, for small $\delta>0$, there exists a $\;M(\delta)\in\mathbb{N}$ such that for all $m\geq M(\delta)$, $G_{u}(\psi_{m})\leq c\;\delta^{2}$ for some constant $c$ and $G_{u}$ is the functional based on $u$.

Proof.

For $u$, $\tilde{u}$ and any $\psi_{m}\in\mathcal{L}^{2}[a,b]$, we have $G_{u}(\psi_{m})=\lvert\lvert u^{\prime}-u_{\psi_{m}}^{\prime}\rvert\rvert_{\mathcal{L}^{2}}^{2}$ and $G_{\tilde{u}}(\psi_{m})=\lvert\lvert\tilde{u}^{\prime}-u_{\psi_{m}}^{\prime}\rvert\rvert_{\mathcal{L}^{2}}^{2}$, then

[TABLE]

and hence from theorem 5.1, the result follows. ∎

5.3. Conditional Well-posedness (Iterative-regularization)

As explained earlier, in an external-parameter based regularization method (like Tikhonov-type regularizations) first, one converts the ill-posed problem to a family of well-posed problem (depending on the parameter value $\beta$) and then, only after finding an appropriate regularization parameter value (say $\beta_{0}$), one proceeds to the recovery process, i.e. completely minimize the corresponding functional $G(.,\beta_{0})$ (as defined in (1.3)). Where as, in a classical iterative regularization method (not involving any external-parameters like $\beta$) one can not recover a regularized solution by simply minimizing a related functional completely, instead stopping the recovery process at an appropriate instance provides the regularizing effect to the solution. That is, one starts to minimize some related functional (to recover the solution) but then terminates the minimization process at an appropriate iteration before it has reached the minimum (to restrict the influence of the noise), i.e. here the iteration stopping index serves as a regularization parameter, for details see [1, 17].

In this section we further explain the above phenomenon by showing that if one attempts to recover the true (or original) solution $\varphi$ by using a noisy data then it will distort the recovery. First we see that for an exact $g$ (or equivalently, an exact $u$) and the functional constructed based on it (i.e. $G_{u}$) we have the true solution ($\varphi$) satisfying $G_{u}(\varphi)=0$. However, for a given noisy $\tilde{g}$, with $\lvert\lvert\tilde{g}-g\rvert\rvert_{\mathcal{L}^{2}}=\delta>0$, and the functional based on it (i.e. $G_{\tilde{u}}$) we will have $G_{\tilde{u}}(\varphi)>0$, see Theorem 5.5. So if we construct a sequence of functions $\psi_{m}^{\delta}\in\mathcal{L}^{2}[a,b]$, using the descent algorithm and based on the noisy data $\tilde{g}$, such that $G_{\tilde{u}}(\psi_{m}^{\delta})\rightarrow 0$ then (from Theorem 5.1) we will have $\psi_{m}^{\delta}\rightarrow\tilde{\varphi}$, where $\tilde{\varphi}$ is the recovered noisy solution satisfying $G_{\tilde{u}}(\tilde{\varphi})=0$. This implies initially $\psi_{m}^{\delta}\rightarrow\varphi$ and then upon further iterations $\psi_{m}^{\delta}$ diverges away from $\varphi$ and approaches $\tilde{\varphi}$. Hence, the errors in the recoveries $\lvert\lvert\psi_{m}^{\delta}-\varphi\rvert\rvert_{\mathcal{L}^{2}}$ follow a semi-convergence nature, i.e. decreases first and then increases. This is a typical behavior of any ill-posed problem and is managed, as stated above, by stopping the descent process at an appropriate iteration $M(\delta)$ such that $G_{\tilde{u}}(\psi_{M(\delta)}^{\delta})>0$ but close to zero (due to the stability Theorems 5.3 and 5.4). Following similar arguments as in (2.6) we can have a lower bound for $G_{\tilde{u}}(\varphi)$.

Theorem 5.5**.**

Given two functions $u$, $\tilde{u}\in\mathcal{H}^{3}_{0}[a,b]$, their respective recovery $\varphi$, $\tilde{\varphi}\in\mathcal{L}^{2}[a,b]$, such that $T_{D}\varphi=-u^{\prime\prime}$ and $T_{D}\tilde{\varphi}=-\tilde{u}^{\prime\prime}$, and let the functional $G_{\tilde{u}}$ be defined based on $\tilde{u}$, that is, $G_{\tilde{u}}(\tilde{\varphi})=0$, then we have

[TABLE]

where $\lambda_{1}=\frac{\pi^{2}}{(b-a)^{2}}$ is the smallest eigenvalue of $-\Delta$ on $\mathcal{H}^{2}_{0}[a,b]$.

Therefore, combining Theorems 5.3 and 5.5 we have the following two sided inequality for $G_{\tilde{u}}(\varphi)$, for some constants $C_{1}$ and $C_{2}$,

[TABLE]

Thus, when $\delta\rightarrow 0$ we have $G_{\tilde{u}}(\varphi)\rightarrow 0$ which implies $\varphi_{\delta}\rightarrow\varphi$ in $\mathcal{L}^{2}[a,b]$. Now though we would like to use the bounds in Theorem 5.5 to terminate the descent process, but we do not known the exact $g$ (or equivalently, the exact $u$), and hence can not use that as the stopping condition. However, if the error norm $\delta=\lvert\lvert g-g_{\delta}\rvert\rvert_{\mathcal{L}^{2}}$ is known then one can use Morozov’s discrepancy principle, [18], as a stopping criterion for the iteration process, that is, terminate the iteration when

[TABLE]

for an appropriate $\tau>1$777In our experiments, we considered $\tau=1$ and the termination condition as $\lvert\lvert T\psi_{m}-g_{\delta}\rvert\rvert_{\mathcal{L}^{2}}<\delta$., and for unknown $\delta$ one usually goes for heuristic approaches to stop the iterations, an example of which is presented in §8.

6. Numerical Implementation

In this section we provide an algorithm to compute the derivative numerically. Notice that though one can use the integral operator equation (1.1) to recover $\varphi$ inversely, for computational efficiency we can further improve the operator and the operator equation, but keeping the theory intact. First we see that the adjoint operator $T_{D}^{*}$ can also provide an integral operator equation of interest,

[TABLE]

where $g_{2}(x)=g(x)-g(b)$ and

[TABLE]

for all $x\in[a,b]\subset\mathbb{R}$. Now we can combine both the operator equations (1.1) and (6.1) to get the following integral operator equation

[TABLE]

where $g_{3}(x)=2g(x)-(g(a)+g(b))$ and for all $x\in[a,b]\subset\mathbb{R}$,

[TABLE]

The advantage of the operator equation (6.2) over (1.1) or (6.1) is that it recovers $\varphi$ symmetrically at the end points. For example if we consider the operator equation (6.1) for the recovery of $\varphi$, then during the descent process the $\mathcal{L}^{2}$-gradient at $\psi$ will be $\nabla_{\mathcal{L}^{2}}^{\psi}G=-T_{D}(-2(u-u_{\psi}))$ (similar to (3.4)) which implies $\nabla_{\mathcal{L}^{2}}^{\psi}G(a)=0$ for all $\psi\in\mathcal{L}^{2}[a,b]$. Hence the boundary data at $a$ for all the evolving $\psi_{m}$’s are going to be invariant during the descent888as explained in the $\mathcal{L}^{2}$-gradient version of the descent algorithm for $G$, in Section 4.. As for (1.1), since $\nabla_{\mathcal{L}^{2}}^{\psi}G(b)=0$, the boundary data at $b$ for all the evolving $\psi_{m}$’s are going to be invariant during the descent. Even though one can opt for the Sobolev gradient of $G$ at $\psi$, $\nabla_{\mathcal{H}^{1}}^{\psi}G$, to counter that problem but, due to the intrinsic decay of the base function $\nabla_{\mathcal{L}^{2}}^{\psi}G$ for all $\psi_{m}$’s at $a$ or $b$, the recovery of $\varphi$ near that respective boundary will not be as good (or symmetric) as at the other end. On the other hand if we use the operator equation (6.2) for the descent recovery of $\varphi$ then the $\mathcal{L}^{2}$-gradient at $\psi$ is going to be

[TABLE]

where

[TABLE]

Thus $\nabla_{\mathcal{L}^{2}}^{\psi}G(a)\neq 0$ and $\nabla_{\mathcal{L}^{2}}^{\psi}G(b)\neq 0$, and hence the recovery of $\varphi$ at both the end points will be performed symmetrically. Now one can derive other gradients, like the Neuberger or conjugate gradient based on this $\mathcal{L}^{2}$-gradient, for the recovery of $\varphi$ depending on the scenarios, that is, based on the prior knowledge of the boundary information (as explained in Section 4).

Corresponding to the operator equation (6.2), the smooth or integrated data $u\in\mathcal{H}^{3}_{0}[a,b]$ will be

[TABLE]

Thus our problem set up now is as follows: for a given $g\in\mathcal{H}^{1}[a,b]$ (and hence a given $u\in\mathcal{H}^{3}_{0}[a,b]$) we want to find a $\varphi\in\mathcal{L}^{2}[a,b]$ such that

[TABLE]

Our inverse approach to achieve $\varphi$ will be to minimize the functional $G$ which is defined, for any $\psi\in\mathcal{L}^{2}[a,b]$, by

[TABLE]

where $u$ is as defined in (6) and $u_{\psi}$ is the solution of the boundary value problem

[TABLE]

Since our new problem set up is almost identical to the old one, the previous theorems and results developed for $T_{D}$ can be similarly extended to $T$. Next we provide a pseudo-code, Algorithm 1, for the descent algorithm described earlier.

Remark 6.1*.*

If prior knowledge of $\varphi(a)$ and $\varphi(b)$ is known then $\psi_{0}$ can be defined as a straight line joining them and we use Dirichlet Neuberger gradient for the descent. If no prior information is known about $\varphi(a)$ and $\varphi(b)$ then we simply choose $\psi_{0}\equiv 0$ and use the Sobolev gradient or conjugate gradient for the descent. It has been numerically seen that having any information of $\varphi(a)$ or $\varphi(b)$ and using it, together with appropriate gradient, significantly improves the convergence rate of the descent process and the efficiency of the recovery. In the examples presented here we have not assumed any prior knowledge of $\varphi(a)$ or $\varphi(b)$ to keep the problem settings as pragmatic as possible.

Remark 6.2*.*

To solve the boundary value problem (4.4) while calculating the Neuberger gradients we used the invariant embedding technique for better numerical results (see [5]). This is very important as this technique enables us to convert the boundary value problem to a system of initial and final value problems and hence one can use the more robust initial value solvers, compared to boundary value solvers, which normally use shooting methods.

Remark 6.3*.*

For all the numerical testings presented in Section 7 we assumed to have prior knowledge on the error norm ($\delta$) and used it as a stopping criteria, as explained in Section 5.3, for the descent process. We also compare the results obtained without any noise information (i.e., using heuristic stopping strategy, see §8) with the results obtained using the noise information, see Tables 2 and 3 in §8.

7. Results

A MATLAB program was written to test the numerical viability of the method. We take an evenly spaced grid with $h=10^{-2}$ in all the examples, unless otherwise specified. In all the examples we used the discrepancy principle (see 5.6) to terminate the iterations when the discrepancy error goes below $\delta$ (which is assumed to be known). In section 8 we discuss the stopping criterion when $\delta$ is unknown.

Example 7.1**.**

**[Comparison with standard regularization methods]

**In this example we compare the inverse recovery using our technique with some of the standard regularization methods. We again perturbed the smooth function $g(x)=\cos(x)$, here we consider the regularity of the data as $\mathcal{H}^{1}$ (i.e., $g\in\mathcal{H}^{(k+1)}[a,b]$ for $k=0$), on $[-.5,.5]$ by random noises to get $\tilde{g}(x)=g(x)+\epsilon(x)$, where $\epsilon$ is a normal random variable with mean [math] and standard deviation $\sigma$. Like in [4] we generated two data sets, one with $h=10^{-2}$ (the dense set) and other with $h=10^{-1}$ (the sparse set). We tested with $\sigma=0.01$ on both the data sets and $\sigma=0.1$ only on the dense set. We compare the relative errors, $\frac{\lvert\lvert\varphi-\tilde{\varphi}\rvert\rvert_{\mathcal{L}^{2}}}{\lvert\lvert\varphi\rvert\rvert_{\mathcal{L}^{2}}}$, obtained in our method (using Neumann $\nabla_{\mathcal{H}^{1}}G$-gradient) with the relative errors provided in [4], which are listed in the Table 1.

Here we can see that our method of numerical differentiation outperforms most of the other methods, in both the dense and sparse situation. Though Tikhonov method performs better for $k=2$, that is when $g$ is assumed to be in $\mathcal{H}^{3}[a,b]$, but for the same smoothness consideration, $g\in\mathcal{H}^{1}[a,b]$ or $k=0$, it fails miserably. In fact, one can prove that the ill-posed problem of numerical differentiation turns out to be well-posed when $g\in\mathcal{H}^{(k+1)}([a,b])$ for $k=1,2$, see [1], which explains the small realtive errors in Tikhonov regularization for $k=1,2$. As stated above the results in Table 1 is obtained using the Sobolev gradient, where as Tables 3 and 2 show a comparison in the recovery errors using different gradients and different stopping criteria, as well as the descent rates associated with them. To compare with the total variation method, which is very effective in recovering discontinuties in the solution, we perform a test on a sharp-edged function similar to the one presented in [4] and the results are shown in Example 7.4. Hence we can consider this method as an universal approach in every scenarios.

In the next example we show that one does not have to assume the normality conditions for the error term, i.e., the assumption that the noise involved should be iid normal random variables is not needed, which is critical for certain other methods, rather it can be a mixture of any random variables.

Example 7.2**.**

**[Noise as a mixture of random variables]

**In the previous example we saw that our method holds out in the presence of large error norm. Here we show that this technique is very effective even in the presence of extreme noise. We perturb the function $g(x)=\sin(x/3)$ on $[0,3\pi]$ to $\tilde{g}(x)=g(x)+\epsilon(x)$999to be consistent with footnote6 we kept $\tilde{g}(a)=g(a)$ and $\tilde{g}(b)=g(b)$, but can be avoided., where $\epsilon$ is the error function obtained from a mixture of uniform($-\delta$, $\delta$) and normal(0, $\delta$) random variables, where $\delta=0.5$. Figure 1 shows the noisy $\tilde{g}$ and the exact $g$, and Figure 2(a) shows the computed derivative $\tilde{\varphi}$ vs. $\varphi(x)=\cos(x/3)/3$. The relative error for the recovery of $\tilde{\varphi}$ is $0.0071$.

In the next example we further pushed the limits by not having a zero-mean error term, which is again crucial for many other methods.

Example 7.3**.**

**[Error with non-zero mean]

**In this example we will show that this method is impressive even when the noise involved has nonzero mean. We consider the settings of the previous example: $g(x)=\sin(x/3)$ on $[0,3\pi]$ is perturbed to $\tilde{g}(x)=g(x)+\epsilon(x)$ but here the error function $\epsilon$ is a mixture of uniform(-0.8$\delta$, 1.2$\delta$) and normal(0.1, $\delta$), for $\delta=0.1$. Figure 2(b) shows the recovery of the derivative $\varphi$ versus the true derivative. The relative error of the recovery for $\varphi$ is around 0.0719.

In the following two examples we provide the results of numerical differentiation done on a piece-wise differentiable functions and compare it with the results obtained in [8] and [4].

Example 7.4**.**

**[Discontinuous source function]

**Here we selected a function randomly from the many functions tested in [8]. The selected function has the following definition:

[TABLE]

The function $y_{2}$ is piece-wise differentiable except at the point $t=0.5$, where it has a sharp edge. The function is then perturbed by a uniform($-\delta$, $\delta$) random variable to get the noisy data $y_{2_{\delta}}$, where we even increased the error norm in our testing from $\delta=0.001$ in [8] to $\delta=$0.01 in our case. Figure 3 shows the recoveries using the method described here and Figure 4(a) shows the result from [8]. We also compare it with a similar result obtained in [4]101010where the test function is $g(x)=|x-0.5|$ on $[0,1]$ and the data set is 100 uniformly distributed points with $\sigma=0.01$. using a total variation regularization method, shown in Figure 4(b).

8. Stopping Criterion II

As explained in §5.3, in the presence noise, one has to terminate the descent process at an appropriate iteration to achieve regularization. The discrepancy principle [21, 22, 20] provides a stopping condition provided the error norm ($\delta$) is known. However, in many of the practical situations it is very hard to determine an estimate of the error norm. In such cases heuristic approaches are taken to determine stopping criteria, such as the L-curve method [24, 25]. In this section we present a new heuristic approach to terminate the iterations when the error norm ($\delta$) is unknown. First we notice that the minimizing functional $G$ used here, as defined in (2.3), does not contain the noisy $\tilde{g}$ directly, rather an integrated (smoothed) version of it ($\tilde{u}$), as compared to a minimizing functional (such as $G_{2}$, defined in (1.3)) used in any standard regularization method. Hence, in addition to avoiding the noisy data from affecting the recovery, the integration process also helps in constructing a stopping strategy, which is explained below. Figure 5 shows the difference in $g$ and $\tilde{g}$ vs. $u$ and $\tilde{u}$, from Example 7.2. We can see, from Figure 5(b), that the integration smooths out the noise present in $\tilde{g}$ to get $\frac{\lvert\lvert\tilde{u}-u\rvert\rvert_{\mathcal{L}^{2}}}{\lvert\lvert u\rvert\rvert_{\mathcal{L}^{2}}}\approx 0.78\%$, where as the noise level in $\tilde{g}$ is $\frac{\lvert\lvert\tilde{g}-g\rvert\rvert_{\mathcal{L}^{2}}}{\lvert\lvert g\rvert\rvert_{\mathcal{L}^{2}}}\approx 55.44\%$. Consequently, the sequence $\{\tilde{g}_{m}:=T\psi_{m}\}_{m\geq 1}$, constructed during the descent process, converges (weakly) in $\mathcal{L}^{2}[a,b]$ to $\tilde{g}$, rather than strongly to $\tilde{g}$, that is, for any $\phi\in\mathcal{L}^{2}[a,b]$ the sequence $\{(\tilde{g}_{m}-\tilde{g},\phi)_{\mathcal{L}^{2}}\}$ converges to zero. In other words, the integration mitigates the effects of the high oscillations originating from the random variable and also of any outliers (as its support is close to zero measure). Also, since the forward operator $T$ (as defined in (6.3)) is smooth, the sequence $\tilde{g}_{m}$ first approximate the exact $g$ (as it is also smooth, $g\in\mathcal{H}^{1}[a,b]$), with the corresponding sequence $\{u_{\psi_{m}}\}$ approximating $\tilde{u}\approx u$, and then the sequence $\{\tilde{g}_{m}\}$ attempts to fit the noisy $\tilde{g}$, which leads to a phenomenon known as overfitting. However, when $\tilde{g}_{m}$ tries to overfit the data (i.e., fit $\tilde{g}$) the sequence values $\lvert\lvert\tilde{u}-u_{\psi_{m}}\rvert\rvert_{\mathcal{L}^{2}}$ increases, since the ovefitting occurs in a smooth fashion (as $T$ is a smooth operator) and, as a result, increases the integral values. This effect can be seen in Figure 6(a), $\lvert\lvert u_{\psi_{m}}-u_{\delta}\rvert\rvert_{\mathcal{L}^{2}}$ descent for Example 7.1 (when $\sigma=0.1$), and in Figure 6(b), $\lvert\lvert T\psi_{m}-g_{\delta}\rvert\rvert_{\mathcal{L}^{2}}$ descent for Example 7.2. One can capture the recoveries at these fluctuating111111the fluctuating occurs since the values of $\lvert\lvert\tilde{g}-g_{m}\rvert\rvert_{\mathcal{L}^{2}}$ tends to decrease first (when approximating the exact $g$) and then increases (when making a transition from $g$ to $\tilde{g}$) and eventually decreases (when trying to fit the noisy $\tilde{g}$, i.e., overfitting) points (of either $\lvert\lvert T\psi_{m}-g_{\delta}\rvert\rvert_{\mathcal{L}^{2}}$, $\lvert\lvert u_{\psi_{m}}^{\prime}-u_{\delta}^{\prime}\rvert\rvert_{\mathcal{L}^{2}}$ or $\lvert\lvert u_{\psi_{m}}-u_{\delta}\rvert\rvert_{\mathcal{L}^{2}}$) and choose the recovery corresponding to the earliest iteration for which $T\psi_{m}$ fits through $g_{\delta}$. Choosing the early fluctuating iteration is especially important when dealing with data with large error level, such as in Example 7.1 ($\sigma=0.1$) and Example 7.2. For example, from Figure 6(b) if one captures the recovery at iteration 4 then the relative error in the recovery is only 8% (see Figure 7(a)). However, even if an appropriate early iteration is not selected, still the recovery errors saturate after certain iterations, rather than blowing up. This is significant when dealing with data having small to moderate error level, such as in Example 7.1 ($\sigma=0.01$), where one can notice (in Figure 7(b)) that the relative errors of the recoveries attain saturation after recovering the optimal solution, since $\lvert\lvert u-u_{\delta}\rvert\rvert_{\mathcal{L}^{2}}\approx 0$ for small $\delta$. Table 2 and 3 shows the relative errors of the recoveries obtained using this heuristic stopping criterion.

Remark 8.1*.*

Note that this phenomena does not occur in Landweber iterations, since one minimizes the functional containing the noisy data $g_{\delta}$ directly, i.e., $G(\psi)=\lvert\lvert T\psi-g_{\delta}\rvert\rvert_{\mathcal{L}^{2}}^{2}$. Figure 8(a) shows the descent of $\lvert\lvert T\psi_{m}-g_{\delta}\rvert\rvert_{\mathcal{L}^{2}}$, when Landweber iterations are implemented for Example 7.1 ($\sigma=0.01$), and Figure 8(b) shows the corresponding descent of the relative errors of the recovered solutions, where can see no fluctuations in Figure 8(a) but the semi-convergence in Figure 8(b). Therefore, without any prior knowledge of $\delta$ it is hard to stop the descent process and avoids the ill-posedness. Where as, notice the saturation of the relative errors of the recovery (Figures 7(b)) when we implement our method to the same problem.

9. Conclusion and Future Research

This algorithm for numerical differentiation is very effective, even in the presence of extreme noise, as can be seen from the examples presented in Section 7. Furthermore, it serves as a universal method to deal with all scenarios such as when the data set is dense or sparse and when the function $g$ is smooth or not smooth. The key feature in this technique is that we are able to upgrade the working space of the problem from $\mathcal{H}^{1}[a,b]$ to $\mathcal{H}^{3}_{0}[a,b]$, which is a much smoother space. Additionally, this method also enjoys many advantages of not encountering the involvement of an external regularization parameter, for example one does not have to determine the optimum parameter choice to balance between the fitting and smoothing of the inverse recovery. Even the heuristic approach for the stopping criteria also provides us with a much better recovery, and hence it’s very applicable in the absence of the error norm.

In a follow up paper, we improve this method to calculate derivatives of functions in higher dimensions and for higher order derivatives. Moreover, we can extend this method to encompass any linear inverse problems and thereby generalize the theory; which will be presented in the coming paper where we will apply this method to recover solution of Fredholm Integral Equations, like deconvolution or general Volterra equation.

Acknowledgment

I am very grateful to Prof. Ian Knowles for his support, encouragement and stimulating discussions throughout the preparation of this paper.