A note on optimization in $\mathbb{R}^n$

TL;DR

This paper introduces an algorithm for constrained optimization in n-dimensional real space, utilizing fundamental analysis tools and the Banach fixed point theorem to establish its main results.

Contribution

It presents a novel algorithm for constrained optimization in imensional real space based on fixed point theory and basic analysis techniques.

Findings

Algorithm effectively solves constrained optimization problems.

Uses Banach fixed point theorem for proof of convergence.

Provides a theoretical foundation for the algorithm's validity.

Abstract

In this article, we develop an algorithm suitable for constrained optimization in $\mathbb{R}^n$. The results are developed through standard tools of n-dimensional real analysis and basic concepts of optimization. Indeed, the well known Banach fixed point theorem has a fundamental role in the main result establishment.

Equations

$$L(x,\lambda )=f(x)+{\lambda }^{2}g(x).$$

$$L_p(x,\lambda ,x_k)=f(x)+{\lambda }^{2}g(x)+\frac{K}{2}|x-x_k{|}^2.$$

$${\tilde{L}}_p(x,\lambda ,x_k)$$

$$\frac{\partial {\tilde{L}}_p(x,\lambda ,x_k)}{\partial x}=0$$

$$f^{\prime \prime }(x_k)(x-x_k)+K(x-x_k)+f^{\prime }(x_k)+{\lambda }^2{g}^{\prime }(x_k)=0,$$

$$x-x_k=-(f^{\prime \prime }(x_k)+K{I}_d{)}^{-1}(f^{\prime }(x_k)+{\lambda }^2{g}^{\prime }(x_k)),$$

$$x(\lambda ,x_k)=x_k-(f^{\prime \prime }(x_k)+K{I}_d{)}^{-1}(f^{\prime }(x_k)+{\lambda }^2{g}^{\prime }(x_k)),$$

$$L_1(\lambda ,x_k)$$

$$\frac{\partial L_1(\lambda ,x_k)}{\partial \lambda }=0,$$

$$[(f^{\prime \prime }(x_k)+K{I}_d{)}^{-1}(f^{\prime }(x_k)+{\lambda }^2{g}^{\prime }(x_k))]\cdot g^{\prime }(x_k)\lambda -\lambda g(x_k)=0,$$

$${\lambda }_1=0$$

$$({\lambda }_2^1{)}^2(x_k)=-\left(\frac{[(f^{\prime \prime }(x_k)+K{I}_d{)}^{-1}{f}^{\prime }(x_k)]\cdot g^{\prime }(x_k)-g(x_k)}{[(f^{\prime \prime }(x_k)+K{I}_d{)}^{-1}{g}^{\prime }(x_k)]\cdot g^{\prime }(x_k)}\right).$$

$${\lambda }^2(x_k)=\max \{0,({\lambda }_2^1{)}^2(x_k)\}.$$

$$|{\lambda }^2(x)-{\lambda }^2(y)|\le {\hat{K}}_3|x-y|,$$

$$x_1=x_0-(f^{\prime \prime }(x_0)+K{I}_d{)}^{-1}(f^{\prime }(x_0)+{\lambda }^2(x_0)g^{\prime }(x_0)),$$

$$x_{k+1}=x_k-(f^{\prime \prime }(x_k)+K{I}_d{)}^{-1}(f^{\prime }(x_k)+{\lambda }^2(x_k)g^{\prime }(x_k)),\forall k\in \mathbb{N}.$$

$$g(x_0)<0$$

$$K_3={\hat{K}}_3\left(\underset{x\in B_r(x_0)}{\sup }|g^{\prime }(x)|\right),$$

$$f^{\prime \prime }(x)+{\lambda }^2(y)g^{\prime \prime }(x)\ge {\alpha }_1({\hat{K}}_1+K)I_d\forall x,y\in B_r(x_0).$$

$$0<{\alpha }_1<1,$$

$$\left(1-\frac{{\alpha }_1}{4}\right)I_d\le ((f^{\prime \prime }(x)+K{I}_d{)}^{-1})(f^{\prime \prime }(y)+K{I}_d)\equiv H(x,y)\le \left(1+\frac{{\alpha }_1}{4}\right)I_d,$$

$$0\le \frac{f^{\prime \prime }(x)+\lambda (y{)}^2{g}^{\prime \prime }(x)}{K-{\hat{K}}_1}\le \left(1-\frac{{\alpha }_1}{2}\right)I_d,\forall x,y\in B_r(x_0).$$

$$0\le (K-\widehat{K_1})I_d\le f^{\prime \prime }(x)+K{I}_d,$$

$$(f^{\prime \prime }(x)+K{I}_d{)}^{-1}\le \frac{1}{K-\widehat{K_1}}{I}_d,$$

$$|(f^{\prime \prime }(x)+K{I}_d{)}^{-1}|K_3\le \frac{K_3}{|K-{\hat{K}}_1|}=\frac{{\alpha }_1}{2},\forall x\in B_r(x_0).$$

$$x_1\in B_{r(1-{\alpha }_0)}(x_0)$$

$$x_2,\dots ,x_{k+1}\in B_r(x_0).$$

$$x_{k+2}-x_{k+1}=-(f^{\prime \prime }(x_{k+1})+K{I}_d{)}^{-1}(f^{\prime }(x_{k+1})+{\lambda }^2(x_{k+1})g^{\prime }(x_{k+1})),$$

$$x_{k+1}-x_k=-(f^{\prime \prime }(x_k)+K{I}_d{)}^{-1}(f^{\prime }(x_k)+{\lambda }^2(x_k)g^{\prime }(x_k)),$$

$$(f^{\prime \prime }(x_{k+1})+K{I}_d)(x_{k+2}-x_{k+1})=-(f^{\prime }(x_{k+1})+{\lambda }^2(x_{k+1})g^{\prime }(x_{k+1})),$$

Full text

††journal: my journal

1 Introduction

In this short letter we develop a proximal algorithm for constrained optimization.

Let $f:\mathbb{R}^{n}\rightarrow\mathbb{R}$ be a $C^{2}$ class function. Consider the problem of minimizing locally $f$ subject to $g(x)\leq 0,$ where $g:\mathbb{R}^{n}\rightarrow\mathbb{R}$ is a given $C^{2}$ class function.

The lagrangian for this problem, denoted by $L:\mathbb{R}^{n+1}\rightarrow\mathbb{R}$ may be expressed by

[TABLE]

We define the proximal formulation for such a problem, denoted by $L_{p}$ by

[TABLE]

2 The main result

Linearizing $L_{p}$, we propose the following procedure for looking for a critical point of such a function:

Consider

[TABLE]

Hence from

[TABLE]

we obtain,

[TABLE]

that is,

[TABLE]

and therefore

[TABLE]

where $I_{d}$ denotes the $n\times n$ identity matrix.

We define $L_{1}(\lambda,x_{k})=\tilde{L}_{p}(x(\lambda,x_{k}),x_{k},\lambda)$ so that

[TABLE]

From

[TABLE]

we get

[TABLE]

so that we have two solutions,

[TABLE]

and

[TABLE]

Observe that if $(\lambda_{2}^{1})^{2}(x_{k})<0$ then $\lambda_{2}^{1}(x_{k})$ is complex so that, from the condition $\lambda^{2}\geq 0$, we obtain

[TABLE]

Also, from the generalized inverse function theorem $\lambda^{2}(x)$ is locally Lipschtzian (see 16 ; 28 ; 12 ; 19 for details). Hence, we may infer that for a given $x_{0}\in\mathbb{R}^{n}$ there exists $r>0$ and $\hat{K}_{3}>0$ such that

[TABLE]

$\forall x,y\in B_{r}(x_{0})$. With such results in mind, for such an $x_{0}\in\mathbb{R}^{n}$, define $\{x_{k}\}$ by

[TABLE]

[TABLE]

Assume

[TABLE]

and there exists $\hat{K}_{1}$ such that $|f^{\prime\prime}(x)|\leq\hat{K}_{1},\;\forall x\in B_{r}(x_{0}).$

Define

[TABLE]

$\alpha_{1}=2K_{3}/|K-\hat{K}_{1}|$ and suppose

[TABLE]

Suppose also $K$ is such that $K>\hat{K}_{1}$,

[TABLE]

[TABLE]

$\forall x,y\in B_{r}(x_{0})$ and

[TABLE]

Observe that since $|f^{\prime\prime}(x)|\leq\hat{K}_{1}$, we have

[TABLE]

so that

[TABLE]

and

[TABLE]

Assume $K>0$ is such that

[TABLE]

and suppose the induction hypotheses

[TABLE]

where $0<\alpha_{0}<1$ is specified in the next lines.

Note that,

[TABLE]

and

[TABLE]

so that,

[TABLE]

and

[TABLE]

Therefore,

[TABLE]

where $\tilde{x}_{k}$ is on the line connecting $x_{k}$ and $x_{k+1}.$

Thus,

[TABLE]

so that

[TABLE]

Observe that, from (5),

[TABLE]

so that

[TABLE]

Hence, from this, (6), (8) and (7), we obtain

[TABLE]

and therefore,

[TABLE]

On the other hand, from (9) we have,

[TABLE]

From (11) and these last two inequalities, we obtain

[TABLE]

Thus, denoting $\alpha_{0}=1-\alpha_{1}/4$, we have obtained,

[TABLE]

so that

[TABLE]

Thus,

[TABLE]

so that

[TABLE]

Hence $x_{k+2}\in B_{r}(x_{0}),$ and therefore the induction is complete, so that,

[TABLE]

Moreover, $\{x_{k}\}$ is a Cauchy sequence, so that there exists $\tilde{x},$ such that

[TABLE]

Finally

[TABLE]

Hence, from this and

[TABLE]

we obtain

[TABLE]

In such a case, from (2) letting $k\rightarrow\infty,$ we also obtain

[TABLE]

Thus if $\tilde{\lambda}^{2}>0$, then $g(\tilde{x})=0.$

If $\tilde{\lambda}=0,$ then $f^{\prime}(\tilde{x})=\mathbf{0}$ and

[TABLE]

so that from (3), since $(f^{\prime\prime}(\tilde{x})+KI_{d})^{-1}$ is positive definite, letting $k\rightarrow\infty$, we get

[TABLE]

That is, in any case,

[TABLE]

Remark 1

For the more general case with $m_{1}$ equality scalar constraints

[TABLE]

and $m_{2}$ inequality scalar constraints

[TABLE]

where $h_{j},g_{l}:\mathbb{R}^{n}\rightarrow\mathbb{R}$ are $C^{2}$ class functions, $\forall j\in\{1,\ldots,m_{1}\}$ and $\forall l\in\{1,\ldots,m_{2}\},$ we assume $m_{1}+m_{2}<n$ and define the Lagrangian $L_{p}$ by

[TABLE]

Linearizing $L_{p}$, we propose the following procedure for looking for a critical point of such a function:

Consider

[TABLE]

Hence from

[TABLE]

we obtain,

[TABLE]

that is,

[TABLE]

and therefore

[TABLE]

where $I_{d}$ denotes the $n\times n$ identity matrix.

We define $L_{1}(\lambda,x_{k})=\tilde{L}_{p}(x(\lambda,x_{k}),x_{k},\lambda),$ so that

[TABLE]

From

[TABLE]

we get

[TABLE]

From

[TABLE]

we have

[TABLE]

$\forall j\in\{1,\ldots,m_{1}\}.$ Solving the linear system which comprises these last $m_{1}$ equations and the $m_{2}$ equations

[TABLE]

$\forall l\in\{1,\ldots,m_{2}\},$ we may obtain a solution

[TABLE]

Thus, to obtain a concerning critical point, we follow the following algorithm.

Choose $x_{0}\in\mathbb{R}^{n}$, $K_{max}\in\mathbb{N}$ ($K_{max}$ is the maximum number of iterations), set $k=0$ and $e_{1}\approx 10^{-5}$. 2. 2.

Obtain a solution

[TABLE]

by solving the linear system (in $(\lambda_{h})_{j}$ and $(\lambda_{g})_{l}^{2}$) indicated in (1) and (1).

Observe that if $(\lambda_{g}^{1})_{l}^{2}<0$ then $(\lambda_{g}^{1})_{l}$ is complex.

To up-date $\lambda_{h}$ and $\lambda_{g}$ proceed as follows: 3. 3.

For each $l\in\{1,\ldots,m_{2}\}$ if $(\lambda_{g})_{l}^{2}(x_{k})\leq 0$, then set $(\lambda_{g})_{l}(x_{k})=0.$ 4. 4.

Define $J=\{l\in\{1,\ldots,m_{2}\}\text{ such that }(\lambda_{g})_{l}^{2}(x_{k})>0\}.$ 5. 5.

Recalculate $(\lambda_{h})_{j}(x_{k})$ and the non-zero $(\lambda_{g})_{l}^{2}(x_{k})\text{ for }l\in J$ through the solution of the linear system (in $(\lambda_{h})_{j}$ and $(\lambda_{g})_{l}^{2}$)

[TABLE]

$\forall j\in\{1,\ldots,m_{1}\}$ and

[TABLE]

$\forall l\in J.$ 6. 6.

If $(\lambda_{g})_{l}^{2}(x_{k})\geq 0,\;\forall l\in\{1,\ldots,m_{2}\}$, then go to 7, otherwise go to item 3. 7. 7.

Up-date $x_{k}$ through the equation

[TABLE] 8. 8.

If $|x_{k+1}-x_{k}|<e_{1}$ or $k>K_{max}$, then stop, otherwise $k:=k+1$ and go to 2.

3 Conclusion

In this article we have developed an algorithm for constrained optimization in $\mathbb{R}^{n}$. We prove the main result only for the special case of a single scalar inequality constraint. However, we highlight the proof of a more general result involving equality and inequality constraints may be developed in a similar fashion, as indicated in remark 1. We postpone the presentation of the formal details for such a more general case for a future work.