On Certain Conditions for Convex Optimization in Hilbert Spaces

TL;DR

This paper extends convex optimization techniques to infinite-dimensional Hilbert spaces, providing optimality conditions and demonstrating their application through a Dirichlet problem example.

Contribution

It introduces first-order optimality conditions for convex problems in Hilbert spaces and applies these results to a specific boundary value problem.

Findings

Optimality condition: $f'(x,d) ext{≥}0$ for all directions at a local solution.

Differentiability implies zero gradient at local minima.

Application to Dirichlet problem demonstrates practical relevance.

Abstract

In this paper convex optimization techniques are employed for convex optimization problems in infinite dimensional Hilbert spaces. A first order optimality condition is given. Let $f : \mathbb{R}^{n}\rightarrow \mathbb{R}$ and let $x\in \mathbb{R}^{n}$ be a local solution to the problem $\min_{x\in \mathbb{R}^{n}} f(x).$ Then $f'(x,d)\geq 0$ for every direction $d\in \mathbb{R}^{n}$ for which $f'(x,d)$ exists. Moreover, Let $f : \mathbb{R}^{n}\rightarrow \mathbb{R}$ be differentiable at $x^{*}\in \mathbb{R}^{n}.$ If $x^{*}$ is a local minimum of $f$, then $\nabla f(x^{*}) = 0.$ A simple application involving the Dirichlet problem is also given.

Equations

$$f(\beta x+(1-\beta )y)\le \beta f(x)+(1-\beta )f(y).$$

$$⟨x-{\varphi }_W(x),x_n-{\varphi }_W(x)⟩\le 0,\forall n.$$

$$\Vert x-{\varphi }_W(x){\Vert }^2$$

$$\lambda \le f(x^{\ast })\le \lim \inf f(x_{n_i})=\lim f(x_{n_i})=\lambda$$

$$f^{\prime }(x,d)=\underset{t\downarrow 0}{\lim }\frac{f(x+td)-f(x)}{t}.$$

$$0\le f^{\prime }(x^{\ast },d)=\nabla f(x^{\ast }{)}^{T}d,$$

Full text

1. Introduction

Studies on optimization has attracted the attention of many mathematicians and researchers over along period of time(see [2], [3], [5], [6], [8] and the references there in). In this paper, we are concerned with the classical results on optimization of convex functionals in infinite-dimensional real Hilbert spaces. When working with infinite-dimensional spaces, a basic difficulty is that, unlike the case in finite-dimension, being closed and bounded does not imply that a set is compact[1]. In reflexive Banach spaces, this problem is mitigated by working in weak topologies and using the result that the closed unit ball is weakly compact[10]. This in turn enables mimicking some of the same ideas in finite-dimensional spaces when working on unconstrained optimization problems[4]. It is the goal of this paper to provide a concise coverage of the problem of minimization of a convex function on a Hilbert space. The focus is on real Hilbert spaces, where there is further structure that makes some of the arguments simpler. Namely, proving that a closed and convex set is also weakly sequentially closed can be done with an elementary argument, whereas to get the same result in a general Banach space we need to invoke Mazurs Theorem[7]. The ideas discussed in this brief note are of great utility in theory of PDEs, where weak solutions of problems are sought in appropriate Sobolev spaces[9]. After a brief review of the requisite preliminaries, we develop the main results. Though, the results in this note are classical, we provide proofs of key theorems for a self contained presentation. A simple application, regarding the Dirichlet problem, is provided for the purposes of illustration. Before moving further we recall an important point about notions of compactness and sequential compactness in weak topologies. It is common knowledge that compactness and sequential compactness are equivalent in metric spaces. The situation is not obvious in the case of weak topology of an infinite-dimensional normed linear space[6].

2. Preliminaries

Definition 2.1**.**

A sequence $x_{n}$ in a Banach space $B$ is said to converge to $x\in B$ if $\lim_{n\rightarrow\infty}x_{n}=x.$ Also a sequence $x_{n}$ in a Hilbert space $H$ converges weakly to $x$ if, $\lim_{n\rightarrow\infty}\langle x_{n},u\rangle=\langle x,u\rangle,\;\forall u\in H.$ We use the notation $x_{n}\rightharpoonup x$ to mean that $x_{n}$ converges weakly to $x.$

Definition 2.2**.**

A real valued function $f$ on a Banach space $B$ is lower semi-continuous (LSC) if $f(x)\leq\lim\inf_{n\rightarrow\infty}f(x_{n})$ for all sequences $x_{n}$ in $B$ such that $x_{n}\rightarrow x$ (strongly) and weakly sequentially lower semi-continuous (weakly sequentially LSC) if $x_{n}\rightharpoonup x.$

Definition 2.3**.**

A non-empty set $W$ is said to be convex if for all $\beta\in[0,1]$ and $\forall\;x,y\in W$ $\beta x+(1-\beta)y\in W.$ Let $X$ be a metric space and $W\subseteq X$ a non-empty convex set. A function $f:W\rightarrow\mathbb{R}$ is convex if for all $\beta\in[0,1]$ and $\forall\;x,y\in W$

[TABLE]

Remark 2.4*.*

We note that the function $f$ in the above definition is called strictly convex if the above inequality is strict for $x\neq y$ and $\beta\in(0,1).$ A function $f$ is convex if and only if its epigraph, $epi(f)$, is convex whereby $epi(f):=f(x,r)\in dom(f)\times\mathbb{R}:f(x)\leq r.$ An optimization problem is convex if both the objective function and feasible set are convex(see[6] for details).

Definition 2.5**.**

Let $\mathbb{R}^{n}$ be an $n$-dimensional real space and $W\subseteq\mathbb{R}^{n}$. A point $x^{*}\in\mathbb{R}^{n}$ is called a global minimizer of the optimization problem $\min_{x\in W}f(x),$ if $x^{*}\in W$ and $f(x^{*})\leq f(x),$ for all $x\in W.$

Definition 2.6**.**

Let $\mathbb{R}^{n}$ be an $n$-dimensional real space and $W\subseteq\mathbb{R}^{n}$. A point $x^{*}\in\mathbb{R}^{n}$ is called a local minimizer of the optimization problem $\min_{x\in W}f(x),$ if there exists a neighbourhood $N$ of $x^{*}$ such that $x^{*}$ is a global minimizer of the problem $\min_{x\in W\cap N}f(x).$ That is there exists $\varepsilon>0$ such that $f(x^{*})\leq f(x),$ whenever $x^{*}\in W$ satisfies $\|x^{*}-x\|\leq\varepsilon.$

Remark 2.7*.*

Any local minimizer of a convex optimization problem is a global minimizer[2].

Proposition 2.8**.**

*Let $B$ be a Banach space and $f:B\rightarrow\mathcal{\mathbb{R}}.$ Then the following are conditions [3] equivalent. (i). $f$ is (weakly sequentially) LSC.

(ii). $epi(f),$ is (weakly sequentially) closed.

Remark 2.9*.*

$f:B\rightarrow\mathcal{\mathbb{R}}$ is coercive if for all $x\in B,$ $\lim_{\|x\|\rightarrow\infty}f(x)=\infty.$

Proposition 2.10**.**

Let $H$ be an infinite dimensional real separable Hilbert space and let $W\subseteq H$ be a (strongly) closed and convex set. Then, $W$ is weakly sequentially closed.

Proof.

Let the sequence $x_{n}\rightharpoonup x$ be in $W.$ It only suffices to show that $x\in W$ by showing that $x=\phi_{W}(x),$ where $\phi_{W}(x)$ is the projection of $x$ into the closed convex set $W$. Indeed, we know that the projection $\phi_{W}(x)$ satisfies the variational inequality, $\langle x-\phi_{W}(x),y-\phi_{W}(x)\rangle\leq 0,$ for all $y\in W.$

So,

[TABLE]

But, $x_{n}\rightharpoonup x$ be in $W$ so we have,

[TABLE]

Hence, by Equation 2.1 we have $\|x-\phi_{W}(x)\|=0.$ That is, $x=\phi_{W}(x).$ ∎

Lemma 2.11**.**

Let $f:H\rightarrow\mathbb{R}$ be a LSC convex function. Then $f$ is weakly LSC.

Proof.

We know that $f$ is convex iff $epi(f)$ is convex. Moreover, $epi(f)$ is strongly closed because $f$ is (strongly) LSC. By Proposition 2.10 we have that $epi(f)$ is weakly sequentially closed implying that $f$ is weakly sequentially LSC.

∎

3. Main Results

Theorem 3.1**.**

Let $H$ be an infinite dimensional real separable Hilbert space and $W\subseteq H$ be a weakly sequentially closed and bounded set. Let $f:W\rightarrow\mathbb{R}$ be weakly sequentially LSC. Then $f$ is bounded from below and has a minimizer on $W$.

Proof.

The proof has two steps:

(i). $f$ is bounded below.

(ii). There exists a minimizer in $W.$

Step(i): Suppose that $f$ is not bounded from below. Then there exist a sequence $x_{n}\in W$ such that $f(x_{n})<-n$ for all $n.$ But $W$ is bounded so $x_{n}$ has a weakly convergent subsequence $x_{n_{i}}$ Furthermore, $W$ is weakly sequentially closed therefore $x\in W$. Then, since f is weakly sequentially LSC we have $f(x)\leq\lim\inf_{n\rightarrow\infty}f(x_{n_{i}})=-\infty$ which is a contradiction. Hence, $f$ is bounded from below.

Step(ii): Let $x_{n}\in W$ be a minimizing sequence for $f$ that is $f(x_{n})\rightarrow\inf_{W}f(x).$ Let $\lambda:=\inf_{W}f(x).$ Since $W$ is bounded and weakly sequentially closed, it follows by [8] that $x_{n}$ has a weakly convergent subsequence has a weakly convergent subsequence $x_{n_{i}}\in W$. But $f$ is weakly sequentially LSC so we have

[TABLE]

So, $f(x^{*})=\lambda$ ∎

Corollary 3.2**.**

Let $H$ be an infinite dimensional real separable Hilbert space and $W\subseteq H$ be a weakly sequentially closed and bounded set. Let $f:W\rightarrow\mathbb{R}^{n}$ be non-empty and closed, and that $f:W\rightarrow\mathbb{R}^{n}$ is LSC and coercive. Then the optimization problem $\inf_{x\in W}f(x)$ admits at least one global minimizer.

Proof.

By [2] with an analogy to the proof of Theorem 3.1 the proof of coercivity is sufficient. ∎

Theorem 3.3**.**

A function that is strictly convex on $W$ has a unique minimizer on W.

Proof.

Assume the contrary, that f $(x)$ is convex yet there are two points $x,y\in W$ such that $f(x)$ and $f(y)$ are local minima. Because of the convexity of $W$ every point on the secant line $\beta x+(1-\beta)y$ is in $W.$ Without loss of generality suppose $f(x)\geq f(y)$ if this is not the case, simply relabel the points. We then have $\beta f(x)+(1-\beta)f(y)<f(y),\forall\;\beta\in(0,1).$ But $f$ is strictly convex, we also have $f(\beta x+(1-\beta)y)<f(x),\forall\;\beta\in(0,1).$ Taking $\beta$ arbitrarily close to [math] along the secant line, $z=\beta x+(1-\beta)y$ remains in $W$(since $W$ is convex) and $f(z)$ remains strictly below $f(x)$ (because $f$ is strictly convex). Therefore, there is no open ball $B$ containing $x$ such that $f(x)<f(z),\forall\;z(B\cap W)\setminus x.$ Therefore, $x$ is not a local minimizer, which is a contradiction. ∎

In this last part we give an optimality conditions. We give the first order condition for optimality here. Consider the function $\psi:\mathbb{R}\rightarrow\mathbb{R}$ given by $\psi(t)=f(x+td)$ for some choice of $x$ and $d$ in $\mathbb{R}^{n}$. The key variational object in this context is the directional derivative of $f$ at a point $x$ in the direction $d$ given by

[TABLE]

When $f$ is differentiable at the point $x\in\mathbb{R}^{n}$, then $f^{\prime}(x,d)=\nabla f(x)^{T}d=\psi^{\prime}(0).$ The next two results give us an optimality condition.

Proposition 3.4**.**

Let $f:\mathbb{R}^{n}\rightarrow\mathbb{R}$ and let $x\in\mathbb{R}^{n}$ be a local solution to the problem $\min_{x\in\mathbb{R}^{n}}f(x).$ Then $f^{\prime}(x,d)\geq 0$ for every direction $d\in\mathbb{R}^{n}$ for which $f^{\prime}(x,d)$ exists.

Theorem 3.5**.**

Let $f:\mathbb{R}^{n}\rightarrow\mathbb{R}$ be differentiable at $x^{*}\in\mathbb{R}^{n}.$ If $x^{*}$ is a local minimum of $f$, then $\nabla f(x^{*})=0.$

Proof.

We know that every differentiable function is continuous so by Proposition 3.4 we have we have

[TABLE]

for all $d\in\mathbb{R}^{n}$. Taking $d=-\nabla f(x^{*})$ we obtain $0\leq-\nabla f(x^{*})^{T}\nabla f(x^{*})=-\|\nabla f(x^{*})\|^{2}\leq 0.$ Therefore, $\nabla f(x^{*})=0.$ ∎

Example 3.6**.**

Consider the Dirichlet problem: $-\triangle u=f,$ in $W$ and $u=0,$ on $\partial W,$ where $W\subset\mathbb{R}^{n}$ is a bounded domain, and $f\in L^{2}(W).$ It is well known that this problem has a weak solution weak which is convex and continuous, and coercive. Thus, the existence of a unique minimizer is ensured by application of Theorem 3.5.

4. Conclusion

With regard to Portfolio Optimization, this study is geared towards applications to particularly Stochastic optimization with consideration to: Cox-Ross-Rubinstein model and Hamilton-Jacobi-Bellman Equation[3].

Acknowledgement. The author is thankful to NRF, Kenya for the financial support no NRF/JOOUST/2016/2017-001 towards this research.