A primal-dual dynamical approach to structured convex minimization problems

TL;DR

This paper introduces a primal-dual dynamical system for structured convex minimization, proving convergence to saddle points and deriving convergence rates, leading to a new numerical algorithm combining proximal methods.

Contribution

It presents a novel dynamical system approach for structured convex problems and derives an explicit discretization that results in an effective numerical algorithm.

Findings

Trajectories asymptotically converge to saddle points.

Convergence rates for feasibility violation and objective function.

Discretization yields a new algorithm combining proximal methods.

Abstract

In this paper we propose a primal-dual dynamical approach to the minimization of a structured convex function consisting of a smooth term, a nonsmooth term, and the composition of another nonsmooth term with a linear continuous operator. In this scope we introduce a dynamical system for which we prove that its trajectories asymptotically converge to a saddle point of the Lagrangian of the underlying convex minimization problem as time tends to infinity. In addition, we provide rates for both the violation of the feasibility condition by the ergodic trajectories and the convergence of the objective function along these ergodic trajectories to its minimal value. Explicit time discretization of the dynamical system results in a numerical algorithm which is a combination of the linearized proximal method of multipliers and the proximal ADMM algorithm.

Equations

$$\underset{x\in \mathcal{H}}{\inf }f(x)+h(x)+g(Ax),$$

$$\underset{\stackrel{(x,z)\in \mathcal{H}\times \mathcal{G}}{Ax-z=0}}{\inf }f(x)+h(x)+g(z).$$

$$l:\mathcal{H}\times \mathcal{G}\times \mathcal{G}⟶\overline{\mathbb{R}},l(x,z,y)=f(x)+h(x)+g(z)+⟨y,Ax-z⟩.$$

$$l(x^{\ast },z^{\ast },y)\le l(x^{\ast },z^{\ast },y^{\ast })\le l(x,z,y^{\ast }),\forall (x,z,y)\in \mathcal{H}\times \mathcal{G}\times \mathcal{G}.$$

$$\underset{y\in \mathcal{G}}{\sup }\left(-(f^{\ast }\square h^{\ast })(-A^{\ast }y)-g^{\ast }(y)\right).$$

$$f^{\ast }:\mathcal{H}\to \overline{\mathbb{R}},f^{\ast }(u)=\underset{x\in \mathcal{H}}{\sup }(⟨u,x⟩-f(x)),h^{\ast }:\mathcal{H}\to \overline{\mathbb{R}},h^{\ast }(u)=\underset{x\in \mathcal{H}}{\sup }(⟨u,x⟩-h(x)),$$

$$g^{\ast }:\mathcal{G}\to \overline{\mathbb{R}},g^{\ast }(y)=\underset{z\in \mathcal{G}}{\sup }(⟨y,z⟩-g(z)),$$

$$(f^{\ast }\square h^{\ast })(x)=\underset{y\in \mathcal{H}}{\inf }(f^{\ast }(y)+h^{\ast }(x-y)).$$

$$\left\{\begin{array}[]{l}0\in\partial f(x)+\nabla h(x)+A^{*}y\\ Ax=z,Ax\in\partial g^{*}(y).\end{array}\right.$$

$$0\in \mathrm{sqri}(\mathrm{dom}g-A(\mathrm{dom}f))$$

$$\mathrm{sqri}Q:=\{x\in Q:{\cup }_{\lambda >0}\lambda (Q-x)\text{mbox}isaclosedlinearsubspaceof\mathcal{G}\}$$

$$\Vert x{\Vert }_U^2=⟨x,Ux⟩\forall x\in \mathcal{H}.$$

$$U_1\succcurlyeq U_2\iff \Vert x{\Vert }_{U_1}^2\ge \Vert x{\Vert }_{U_2}^2\forall x\in \mathcal{H}.$$

$$P_{\alpha }(\mathcal{H})=\{U\in S_{+}(\mathcal{H}):U\succcurlyeq \alpha I\},$$

$$\left\{\begin{array}[]{llll}\dot{x}(t)+x(t)\in\left(\partial f+cA^{*}A+M_{1}(t)\right)^{-1}\left(M_{1}(t)x(t)+cA^{*}z(t)-A^{*}y(t)-{\nabla}h(x(t))\right)\\ \\ \dot{z}(t)+z(t)\in\left(\partial g+cI+M_{2}(t)\right)^{-1}\left(M_{2}(t)z(t)+cA(\gamma\dot{x}(t)+x(t))+y(t)\right)\\ \\ \dot{y}(t)=cA(x(t)+\dot{x}(t))-c(z(t)+\dot{z}(t))\\ \\ x(0)=x^{0}\in{\mathcal{H}},\,z(0)=z^{0}\in{\mathcal{G}},\,y(0)=y^{0}\in{\mathcal{G}},\end{array}\right.$$

$$(Cstrong)\text{there exists}\alpha >0\text{such that}c{A}^{\ast }A+M_1(t)\in P_{\alpha }(\mathcal{H})\forall t\in [0,+\infty ).$$

$$0\in \partial f(\dot{x}(t)+x(t))+c{A}^{\ast }A(\dot{x}(t)+x(t))+M_1(t)\dot{x}(t)-(c{A}^{\ast }z(t)-A^{\ast }y(t)-\nabla h(x(t)))\forall t\in [0,+\infty ),$$

$$0\in \partial g(\dot{z}(t)+z(t))+c(\dot{z}(t)+z(t))-cA(\gamma \dot{x}(t)+x(t))-y(t)+M_2(t)\dot{z}(t)\forall t\in [0,+\infty ).$$

$$0\in \frac{1}{c}\partial f(x^{k+1})+A^{\ast }A{x}^{k+1}+\frac{M_1^k}{c}(x^{k+1}-x^k)-A^{\ast }{z}^k+\frac{A^{\ast }}{c}{y}^k+\frac{1}{c}\nabla h(x^k)\forall k\ge 0.$$

$$0\in{\partial}\left(f(x)+\langle x-x^{k},{\nabla}h(x^{k})\rangle+\frac{c}{2}\left\|Ax-z^{k}+\frac{y^{k}}{c}\right\|^{2}+\frac{1}{2}\|x-x^{k}\|^{2}_{M_{1}^{k}}\right)\bigg{|}_{x=x^{k+1}}$$

$$x^{k+1}\in \underset{x\in \mathcal{H}}{\mathrm{argmin}}\left(f(x)+⟨x-x^k,\nabla h(x^k)⟩+\frac{c}{2}{\Vert Ax-z^k+\frac{y^k}{c}\Vert }^2+\frac{1}{2}\Vert x-x^k{\Vert }_{M_1^k}^2\right).$$

$$0\in{\partial}\left(g(z)+\frac{c}{2}\left\|A(\gamma x^{k+1}+(1-\gamma)x^{k})-z+\frac{y^{k}}{c}\right\|^{2}+\frac{1}{2}\|z-z^{k}\|^{2}_{M_{2}^{k}}\right)\bigg{|}_{z=z^{k+1}},$$

$$z^{k+1}\in \underset{z\in \mathcal{G}}{\mathrm{argmin}}\left(g(z)+\frac{c}{2}{\Vert A(\gamma x^{k+1}+(1-\gamma )x^k)-z+\frac{y^k}{c}\Vert }^2+\frac{1}{2}\Vert z-z^k{\Vert }_{M_2^k}^2\right).$$

$$\left\{\begin{array}[]{llll}x^{k+1}\in\operatorname*{argmin}\limits_{x\in\mathcal{H}}\left(f(x)+\langle x-x^{k},{\nabla}h(x^{k})\rangle+\frac{c}{2}\left\|Ax-z^{k}+\frac{y^{k}}{c}\right\|^{2}+\frac{1}{2}\|x-x^{k}\|^{2}_{M_{1}^{k}}\right)\\ \\ z^{k+1}\in\operatorname*{argmin}\limits_{z\in\mathcal{G}}\left(g(z)+\frac{c}{2}\left\|A(\gamma x^{k+1}+(1-\gamma)x^{k})-z+\frac{y^{k}}{c}\right\|^{2}+\frac{1}{2}\|z-z^{k}\|^{2}_{M_{2}^{k}}\right)\\ \\ y^{k+1}=y^{k}+c(Ax^{k+1}-z^{k+1}).\end{array}\right.$$

$$M_1(t)=\frac{1}{\tau (t)}I-c{A}^{\ast }A\text{mbox}and{M}_2(t)=0,$$

$$\left(\frac{1}{\tau (t)}I-c{A}^{\ast }A\right)x(t)+c{A}^{\ast }z(t)-A^{\ast }y(t)-\nabla h(x(t))\in \frac{1}{\tau (t)}\dot{x}(t)+\frac{1}{\tau (t)}x(t)+\partial f(\dot{x}(t)+x(t))$$

$$\dot{x}(t)+x(t)=(I+\tau (t)\partial f{)}^{-1}\left((I-c\tau (t)A^{\ast }A)x(t)+c\tau (t)A^{\ast }z(t)-\tau (t)A^{\ast }y(t)-\tau (t)\nabla h(x(t))\right).$$

$$\dot{x}(t)+x(t)=\operatorname*{prox}\nolimits_{\tau(t)f}\big{(}(I-c\tau(t)A^{*}A)x(t)+c\tau(t)A^{*}z(t)-\tau(t)A^{*}y(t)-\tau(t){\nabla}h(x(t))\big{)},$$

$$\underset{\kappa }{\mathrm{prox}}:\mathcal{H}\to \mathcal{H},\underset{\kappa }{\mathrm{prox}}(x)=\underset{y\in \mathcal{H}}{\mathrm{argmin}}\left\{\kappa (y)+\frac{1}{2}\Vert x-y{\Vert }^2\right\}=(I+\partial \kappa {)}^{-1}(x),$$

$$\dot{y}(t)+y(t)+c(\gamma -1)A\dot{x}(t)\in \partial g(\dot{z}(t)+z(t)),$$

Full text

A primal-dual dynamical approach to structured convex minimization problems

Radu Ioan Boţ University of Vienna, Faculty of Mathematics, Oskar-Morgenstern-Platz 1, A-1090 Vienna, Austria, email: radu.bot@univie.ac.at. Research partially supported by FWF (Austrian Science Fund), project I 2419-N32.

  

Ernö Robert Csetnek University of Vienna, Faculty of Mathematics, Oskar-Morgenstern-Platz 1, A-1090 Vienna, Austria, email: ernoe.robert.csetnek@univie.ac.at. Research supported by FWF (Austrian Science Fund), project P 29809-N32.

  

Szilárd Csaba László Technical University of Cluj-Napoca, Department of Mathematics, Memorandumului 28, Cluj-Napoca, Romania, email: szilard.laszlo@math.utcluj.ro. This work was supported by a grant of Ministry of Research and Innovation, CNCS - UEFISCDI, project number PN-III-P1-1.1-TE-2016-0266, within PNCDI III.

Abstract. In this paper we propose a primal-dual dynamical approach to the minimization of a structured convex function consisting of a smooth term, a nonsmooth term, and the composition of another nonsmooth term with a linear continuous operator. In this scope we introduce a dynamical system for which we prove that its trajectories asymptotically converge to a saddle point of the Lagrangian of the underlying convex minimization problem as time tends to infinity. In addition, we provide rates for both the violation of the feasibility condition by the ergodic trajectories and the convergence of the objective function along these ergodic trajectories to its minimal value. Explicit time discretization of the dynamical system results in a numerical algorithm which is a combination of the linearized proximal method of multipliers and the proximal ADMM algorithm.

Keywords. structured convex minimization, dynamical system, proximal ADMM algorithm, primal-dual algorithm

AMS Subject Classification. 37N40, 49N15, 90C25, 90C46

1 Introduction and preliminaries

For ${\mathcal{H}}$ and ${\mathcal{G}}$ real Hilbert spaces, we consider the convex minimization problem

[TABLE]

where $f:{\mathcal{H}}\longrightarrow\operatorname*{\overline{\mathbb{R}}}=\mathbb{R}\cup\{\pm\infty\}$ and $g:{\mathcal{G}}\longrightarrow\operatorname*{\overline{\mathbb{R}}}$ are proper, convex and lower semicontinuous functions, $h:\mathcal{H}\longrightarrow\mathbb{R}$ is a convex and Fréchet differentiable function with $L_{h}$-Lipschitz continuous gradient $(L_{h}\geq 0)$, i.e. $\|{\nabla}h(x)-{\nabla}h(y)\|\leq L_{h}\|x-y\|$ for every $x,y\in\mathcal{H}$, and $A:{\mathcal{H}}\longrightarrow{\mathcal{G}}$ is a continuous linear operator.

Problem (1) can be rewritten as

[TABLE]

Obviously, $x^{*}\in{\mathcal{H}}$ is an optimal solution of (1) if and only if $(x^{*},z^{*})\in{\mathcal{H}}\times{\mathcal{G}}$ is an optimal solution of (2) and $Ax^{*}=z^{*}$.

Based on this reformulation of problem (1) we define its Lagrangian

[TABLE]

An element $(x^{*},z^{*},y^{*})\in{\mathcal{H}}\times{\mathcal{G}}\times{\mathcal{G}}$ is said to be a saddle point of the Lagrangian $l$, if

[TABLE]

It is known that $(x^{*},z^{*},y^{*})\in{\mathcal{H}}\times{\mathcal{G}}\times{\mathcal{G}}$ is a saddle point of $l$ if and only if $x^{*}$ is an optimal solution of (1), $Ax^{*}=z^{*}$, and $y^{*}$ is an optimal solution of the Fenchel dual to problem (1), which reads

[TABLE]

In this situation the optimal objective values of (1) and (3) coincide.

In the formulation of (3),

[TABLE]

and

[TABLE]

denote the conjugate functions of $f,h$ and $g$, respectively, and $A^{*}:{\cal G}\rightarrow{\cal H}$ denotes the adjoint operator of $A$. The infimal convolution $f^{*}\square h^{*}:\mathcal{H}\to\operatorname*{\overline{\mathbb{R}}}$ of the functions $f^{*}$ and $h^{*}$ is defined by

[TABLE]

It is also known that $(x^{*},z^{*},y^{*})\in{\mathcal{H}}\times{\mathcal{G}}\times{\mathcal{G}}$ is a saddle point of the Lagrangian $l$ if and only if it is a solution of the following system of primal-dual optimality conditions

[TABLE]

We recall that the convex subdifferential of the function $f:{\cal H}\rightarrow\operatorname*{\overline{\mathbb{R}}}$ at $x\in{\cal H}$ is defined by $\partial f(x)=\{u\in{\cal H}:f(x^{\prime})-f(x)\geq\langle u,x^{\prime}-x\rangle\ \forall x^{\prime}\in{\cal H}\}$, for $f(x)\in\mathbb{R}$, and by $\partial f(x)=\emptyset$, otherwise.

A saddle point of the Lagrangian $l$ exists whenever the primal problem (1) has an optimal solution and the so-called Attouch-Brézis regularity condition

[TABLE]

holds. Here,

[TABLE]

denotes the strong quasi-relative interior of a set $Q\subseteq\mathcal{G}$. We refer the reader to [9, 11, 28] for more insights into the world of regularity conditions and convex duality theory.

Let $S_{+}(\mathcal{H})$ denote the family of continuous linear operators $U:\mathcal{H}\longrightarrow\mathcal{H}$ which are self-adjoint and positive semidefinite. For $U\in S_{+}(\mathcal{H})$ we introduce the following seminorm on ${\cal H}$:

[TABLE]

This introduces on $S_{+}(\mathcal{H})$ the following partial ordering: for $U_{1},U_{2}\in S_{+}(\mathcal{H})$

[TABLE]

For $\alpha>0$ fixed, let be

[TABLE]

where $I:\mathcal{H}\longrightarrow\mathcal{H},\,I(x)=x,$ denotes the identity operator on ${\cal H}$.

The subject of our investigations in this paper will be the following dynamical system, for which we will show that it asymptotically approaches the set of solutions of the primal-dual pair of optimization problems (1)-(3)

[TABLE]

where $c>0$, $\gamma\in[0,1]$, and $M_{1}:[0,+\infty)\longrightarrow S_{+}(\mathcal{H})$ and $M_{2}:[0,+\infty)\longrightarrow S_{+}(\mathcal{G})$ are such that

[TABLE]

One of the motivation for the study of this dynamical system comes from the fact that, as we will see in Remark 1, it provides through explicit time discretization a numerical algorithm which is a combination of the linearized proximal method of multipliers and the proximal ADMM algorithm.

In the next section we will show the existence and uniqueness of strong global solutions for the dynamical system (4) in the framework of the Cauchy-Lipschitz Theorem. In Section 3 we will prove some technical results, which will play an important role in the asymptotic analyis. In Section 4 we will investigate the asymptotic behaviour of the trajectories as the time tends to infinity. By carrying out a Lyapunov analysis and by relying on the continuous variant of the Opial Lemma, we are able to prove that the trajectories generated by (4) asymptotically convergence to a saddle point of the Lagrangian $l$. Furthermore, we provide convergence rates of ${\cal O}(\frac{1}{t})$ for the violation of the feasibility condition by ergodic trajectories and the convergence of the objective function along these ergodic trajectories to its minimal value.

The approach of optimization problems by dynamical systems has a long tradition. Crandall and Pazy considered in [20] dynamical systems governed by subdifferential operators (and more general by maximally monotone operators) in Hilbert spaces, addressed questions like the existence and uniqueness of solution trajectories, and related the latter to the theory of semi-groups of nonlinear contractions. Brézis [14] studied the asymptotic behaviour of the trajectories for dynamical systems governed by convex subdifferentials, and Bruck carried out in [15] a similar analysis for maximally monotone operators. Dynamical systems defined via resolvent/proximal evaluations of the governing operators have enjoyed much attention in the last years, as they result by explicit time discretization in relaxed versions of standard numerical algorithms, with high flexibility and good numerical performances. Abbas and Attouch introduced in [1] a forward-backward dynamical system, by extending to more general optimization problems an approach proposed by Antipin in [5] and Bolte in [10] on a gradient-projected dynamical system associated to the minimization of a smooth convex function over a convex closed set. Implicit dynamical systems were considered also in [13] in the context of monotone inclusion problems. A dynamical system of forward-backward-forward type was considered in [7], while, a dynamical system of Douglas-Rachford type was recently introduced in [21].

It is important to notice that the approaches mentioned above have been introduced in connection with the study of “simple” monotone inclusion and convex minimization problems. They rely on straightforward splitting strategies and cannot be efficiently used when addressing structured minimization problems, like (1), which need to be addressed from a primal and a dual perspective, thus, require for tools and techniques from the convex duality theory. The dynamical approach we introduce and investigate in this paper is, to our knowledge, the first meant to address structured convex minimization problems in the spirit of the full splitting paradigm.

Remark 1**.**

The first inclusion in (4) can be equivalently written as

[TABLE]

while the second one as

[TABLE]

The explicit discretization of (5) with respect to the time variable $t$ and constant step equal to $1$ yields the iterative scheme

[TABLE]

By convex subdifferential calculus, one can easily see that this can be for every $k\geq 0$ equivalently written as

[TABLE]

and, further, as

[TABLE]

Similarly, (6) leads for every $k\geq 0$ to

[TABLE]

which is nothing else than

[TABLE]

Here, $(M_{1}^{k})_{k\geq 0}$ and $(M_{2}^{k})_{k\geq 0}$ are two operator sequences in $S_{+}(\mathcal{H})$ and $S_{+}(\mathcal{G})$, respectively.

Thus the dynamical system (4) leads through explicit time discretization to a numerical algorithm, which, for a starting point $(x^{0},z^{0},y^{0})\in{\cal H}\times{\cal G}\times{\cal G}$, generates a sequence $(x^{k},z^{k},y^{k})_{k\geq 0}$ for every $k\geq 0$ as follows

[TABLE]

The algorithm (7) is a combination of the linearized proximal method of multipliers and the proximal ADMM algorithm.

Indeed, in the case when $\gamma=1$, (7) becomes the proximal ADMM algorithm with variable metrics from [8] (see, also, [12]). If, in addition, $h=0$ and the operator sequences $(M_{1}^{k})_{k\geq 0}$ and $(M_{2}^{k})_{k\geq 0}$ are constant, then (7) becomes the proximal ADMM algorithm investigated in [25, Section 3.2] (see, also, [23]). It is known that the proximal ADMM algorithm can be seen as a generalization of the full splitting primal-dual algorithms of Chambolle-Pock (see [16]) and Condat-Vu (see [19, 27]).

On the other hand, in the case when $\gamma=0$, (7) becomes an extension of the linearized proximal method of multipliers of Chen-Teboulle (see [17], [25, Algorithm 1]).

In the following remark we provide a particular choice for the maps $M_{1}$ and $M_{2}$, which transforms (4) into a dynamical system of primal-dual type formulated in the spirit of the full splitting paradigm.

Remark 2**.**

For every $t\in[0,+\infty)$, define

[TABLE]

where $\tau(t)>0$ is such that $c\tau(t)\|A\|^{2}\leq 1$.

Let $t\in[0,+\infty)$ be fixed. In this particular setting, (5) is equivalent to

[TABLE]

and further to

[TABLE]

In other words,

[TABLE]

where

[TABLE]

denotes the proximal point operator of a proper, convex and lower semicontinuous function $\kappa:{\cal H}\rightarrow\operatorname*{\overline{\mathbb{R}}}$.

On the other hand, relation (6) is equivalent to

[TABLE]

hence,

[TABLE]

This is further equivalent to

[TABLE]

and further to

[TABLE]

In other words,

[TABLE]

Consequently, in this particular setting, the dynamical system (4) can be equivalently written as

[TABLE]

Let us also mention that when $h=0$ and $\gamma=1$ the dynamical system (8) reads

[TABLE]

The explicit time discretization of (9) leads to a numerical algorithm, which, for a starting point $(x^{0},z^{0},y^{0})\in{\cal H}\times{\cal G}\times{\cal G}$, generates the sequence $(x^{k},z^{k},y^{k})_{k\geq 0}$ for every $k\geq 0$ as follows

[TABLE]

By substituting in the first equation of (10) the term $cAx^{k}-cz^{k}$ by $y^{k}-y^{k-1}$, which is allowed according to the last equation, one can easily see that (10) is equivalent to the following numerical algorithm, which, for a starting point $(x^{0},y^{0},y^{-1})\in{\cal H}\times{\cal G}\times{\cal G},y^{0}=y^{-1}$, generates the sequence $(x^{k},y^{k})_{k\geq 0}$ for every $k\geq 0$ as follows

[TABLE]

For $\tau_{k}=\tau>0$ for every $k\geq 0$, (11) is nothing else than the primal-dual algorithm proposed by Chambolle and Pock in [16].

Remark 3**.**

The maps $M_{1}:[0,+\infty)\longrightarrow S_{+}(\mathcal{H})$ and $M_{2}:[0,+\infty)\longrightarrow S_{+}(\mathcal{G})$ can be seen as inducing a variable renorming of the underlying Hilbert space and, as seen in the remark above, allow the use of variable step sizes. In addition, they might provide favourable settings for the derivation of convergence rates for function values along the trajectories, as it is the case for the discrete time counterpart of (4) (see [12, Section 3]).

Example 1**.**

We will illustrate the way in which the parameters $\gamma,c$ and $\tau(t),t\in[0,+\infty)$ may influence the asymptotic convergence of the primal and dual trajectories via numerical experiments. In this scope, we considered the following primal optimization problem

[TABLE]

which is in fact problem (1) written in the following particular setting: ${\mathcal{H}}={\mathcal{G}}=\mathbb{R}^{2}$, $f,g,h:\mathbb{R}^{2}\to\mathbb{R}$, $f(x)=\frac{1}{2}\|x\|_{2}^{2}$, $g(x)=\|x\|_{1}$, $h(x)=0$, for every $x\in\mathbb{R}^{2}$, and $A:\mathbb{R}^{2}\to\mathbb{R}^{2}$, $A(x_{1},x_{2})=(x_{1}-x_{2},x_{1}+x_{2})$. Then $\overline{x}=(0,0)$ is the unique optimal solution of (12) and that

[TABLE]

is the Fenchel dual problem of (12). Thus $\overline{y}=(0,0)$ is the unique dual optimal solution.

We considered the dynamical system (8) attached to the primal-dual pair (12)-(13) with starting points $x^{0}=(-10,10)$, $z^{0}=Ax^{0}=(-20,0)$ and $y^{0}=(-10,10)$ in the case when $\tau(t)=\tau>0$ for every $t\in[0,+\infty)$ is a constant function. In order to solve the resulting dynamical system we used the Matlab function ode15s and, to this end, we reformulated it as

[TABLE]

where

[TABLE]

and

[TABLE]

is defined as

[TABLE]

Notice that

[TABLE]

where $\operatorname*{proj}_{Q}$ denotes the projection operator on a convex and closed set $Q\subseteq{\cal H}$.

According to Theorem 12, the asymptotic convergence of the trajectories can be guaranteed when $\tau c\|A\|^{2}\leq 1$. Since $\|A\|=\sqrt{2}$, we considered for $\tau c\in(0,\frac{1}{2})$ three different choices, namely, $\tau c=0.49,0.25$ and $0.01$. The primal and the dual trajectories generated by the dynamical system for each of these three choices are represented in the figures 1, 2 and 3, respectively. The first row of each figure represents the primal trajectories $x(t)$ for $\gamma=0.99,0.5$ and $0.01$, while the second row represents the dual trajectories $y(t)$ for the same choices of the parameter $\gamma$.

One can see that the parameter $\gamma$ has a strong influence on the asymptotic behaviour of the primal and dual trajectories. Namely, in all three figures, thus independently of the choice of the parameters $\tau$ and $c$, the primal and the dual trajectories converge faster to the corresponding primal and dual solutions, respectively, for larger values of $\gamma$, namely, when $\gamma$ is closer to $1$. As $\gamma$ is the coefficient of the derivative in the second inclusion of the dynamical system, it can be seen as a constant which characterizes its level of implicitness. In particular, more implicitness promotes a better asymptotic convergence. On the other hand, we notice that the smaller the values of $\tau c$ are, the smaller is the influence of $\gamma$ on the asymptotic convergence of the trajectories.

Notations**.**

The following two functions will play an important role in particular in the forthcoming analysis

[TABLE]

and

[TABLE]

With these two notations, the dynamical system (4) can be rewritten as

[TABLE]

Let $t\in[0,+\infty)$ be fixed. The function $G(t,\cdot)$ is proper, convex and lower semicontinuous, hence $z\rightarrow G(t,z)+\frac{c}{2}\|z-v\|^{2}$ is proper, strongly convex and lower semicontinuous for every $v\in{\cal G}$. This allows us to use the sign equal in the second relation of (14). On the other hand, a sufficient condition which guarantees that the function $x\mapsto F(x,t)+\frac{c}{2}\left\|x-\left(\frac{1}{c}M_{1}(t)x(t)+A^{*}z(t)-\frac{A^{*}}{c}y(t)-\frac{1}{c}{\nabla}h(x(t))\right)\right\|^{2}$, which is proper and lower semicontinuous, is strongly convex is that there exists $\alpha(t)>0$ such that $cA^{*}A+M_{1}(t)\in P_{\alpha(t)}(\mathcal{H})$. This actually ensures that $x\rightarrow F(t,x)+\frac{c}{2}\|x-u\|^{2}$ is proper, strongly convex and lower semicontinuous for every $u\in{\cal H}$.

This means that if the assumption

[TABLE]

holds, then we can use also in the first relation of (14) the sign equal. It is easy to see, that, if $(Cweak)$ holds, then $\partial f+cA^{*}A+M_{1}(t)$ is $\alpha(t)$-strongly monotone for every $t\in[0,+\infty)$. In other words, for every $t\in[0,+\infty)$, all $u,v\in{\cal H}$ and all $u^{*}\in(\partial f+cA^{*}A+M_{1}(t))(u),x^{*}\in(\partial f+cA^{*}A+M_{1}(t))(x)$ we have

[TABLE]

Notice that, since $A^{*}A\in S_{+}(\mathcal{H})$ and $M_{1}(t)\in S_{+}(\mathcal{H})$ for every $t\in[0,+\infty)$, $(Cweak)$ is fulfilled, if

[TABLE]

or, if

[TABLE]

Notice also that, if ${\cal H}$ is a finite dimensional Hilbert space, then (16), which is independent of $t$, is nothing else than $A^{*}A$ is positively definite or, equivalently, $A$ is injective.

Let $S=\{x\in\mathcal{H}:\|x\|=1\}$ be the unit sphere of $\mathcal{H}$. Assumption $(Cweak)$ is fulfilled if and only if $\inf_{x\in S}\|x\|^{2}_{cA^{*}A+M_{1}(t)}>0$ for every $t\in[0,+\infty)$. In this case we can take $\alpha(t):=\inf_{x\in S}\|x\|^{2}_{cA^{*}A+M_{1}(t)}$ for every $t\in[0,+\infty)$.

Obviously, if $(Cstrong)$ holds, then $(Cweak)$ holds with $\alpha(t):=\alpha>0$ for every $t\in[0,+\infty)$.

2 Existence and uniqueness of the trajectories

In this section we will investigate the existence and uniqueness of the trajectories generated by (4). We start by recalling the definition of a locally absolutely continuous map.

Definition 1**.**

A function $x:[0,+\infty)\rightarrow{\mathcal{H}}$ is said to be locally absolutely continuous, if it is absolutely continuous on every interval $[0,T],\,T>0$; that is, for every $T>0$ there exists an integrable function $y:[0,T]\rightarrow{\mathcal{H}}$ such that

[TABLE]

Remark 4**.**

(a) Every absolutely continuous function is differentiable almost everywhere, its derivative coincides with its distributional derivative almost everywhere and one can recover the function from its derivative $\dot{x}=y$ by the above integration formula.

(b) Let be $T>0$ and $x:[0,T]\rightarrow{\mathcal{H}}$ an absolutely continuous function. This is equivalent to (see [6, 2]): for every $\varepsilon>0$ there exists $\eta>0$ such that for any finite family of intervals $I_{k}=(a_{k},b_{k})\subseteq[0,T]$ the following property holds:

[TABLE]

From this characterization it is easy to see that, if $B:{\mathcal{H}}\rightarrow{\mathcal{H}}$ is $L$-Lipschitz continuous with $L\geq 0$, then the function $z=B\circ x$ is absolutely continuous, too. This means that $z$ is differentiable almost everywhere and $\|\dot{z}(\cdot)\|\leq L\|\dot{x}(\cdot)\|$ holds almost everywhere.

The following definition specifies which type of solutions we consider in the analysis of the dynamical system (4).

Definition 2**.**

Let $(x^{0},z^{0},y^{0})\in{\mathcal{H}}\times{\mathcal{G}}\times{\mathcal{G}}$, $c>0$, $\gamma\in[0,1]$, and $M_{1}:[0,+\infty)\rightarrow{\cal S}_{+}({\cal H})$ and $M_{2}:[0,+\infty)\rightarrow{\cal S}_{+}({\cal G})$. We say that the function $(x,z,y):[0,+\infty)\longrightarrow{\mathcal{H}}\times{\mathcal{G}}\times{\mathcal{G}}$ is a strong global solutions of (4), if the following properties are satisfied:

1. (i)

the functions $x,z,y$ are locally absolutely continuous; 2. (ii)

for almost every $t\in[0,+\infty)$

[TABLE] 3. (iii)

$x(0)=x^{0},\,z(0)=z^{0},\mbox{ and }y(0)=y^{0}.$**

The following results will be useful in the proof of the existence and uniqueness theorem.

Lemma 1**.**

Assume that $(Cweak)$ holds. Then, for every fixed $t\in[0,+\infty)$, the operator

[TABLE]

is Lipschitz continuous.

Proof.

Let $t\in[0,+\infty)$ be fixed and $u,v\in\mathcal{H}$. By subdifferential calculus we obtain that

[TABLE]

and

[TABLE]

Using that, due to $(Cweak)$, $\partial f+cA^{*}A+M_{1}(t)$ is $\alpha(t)$-strongly monotone, we get

[TABLE]

By the Cauchy-Schwartz inequality we obtain

[TABLE]

which shows that $S_{t}$ is Lipschitz continuous with constant $\frac{c}{\alpha(t)}$. ∎

Now we are going to prove another technical result which will be used in the proof of the main theorem of this section.

Lemma 2**.**

Assume that $(Cweak)$ holds. Let be $(x,z,y)\in\mathcal{H}\times\mathcal{G}\times\mathcal{G}$ and the maps $R_{(x,z,y)}:[0,+\infty)\longrightarrow\mathcal{H},$

[TABLE]

and $Q_{(x,z,y)}:[0,+\infty)\longrightarrow\mathcal{G},$

[TABLE]

Then the following statements are true for every $t,r\in[0,+\infty)$:

• (i)

$\|R_{(x,z,y)}(t)-R_{(x,z,y)}(r)\|\leq\frac{\|R_{(x,z,y)}(r)\|}{\alpha(t)}\|M_{1}(t)-M_{1}(r)\|;$**

• (ii)

$\|Q_{(x,z,y)}(t)-Q_{(x,z,y)}(r)\|\leq\frac{\|Q_{(x,z,y)}(r)\|}{c}\|M_{2}(t)-M_{2}(r)\|+\frac{\gamma\|A\|\|R_{(x,z,y)}(r)\|}{\alpha(t)}\|M_{1}(t)-M_{1}(r)\|.$**

Proof.

Let $t,r\in[0,+\infty)$ be fixed.

(i) From the definition of $R_{(x,z,y)}$ one has

[TABLE]

and

[TABLE]

which is equivalent to

[TABLE]

Using again that ${\partial}f+cA^{*}A+M_{1}(t)$ is $\alpha(t)$-strongly monotone for every $t\in[0,+\infty)$, we obtain

[TABLE]

The conclusion follows via the Cauchy-Schwarz inequality.

(ii) From the definition of $Q_{(x,z,y)}$ one has

[TABLE]

and

[TABLE]

which is equivalent to

[TABLE]

Using that ${\partial}g+M_{2}(t)+cI$ is $c-$strongly monotone, we obtain

[TABLE]

From the Cauchy-Schwarz inequality and (i) it follows

[TABLE]

∎

Now we can prove existence and uniqueness of a strong global solution of (4) under $(CStrong)$. To this end we will first reformulate (14) as a particular first order dynamical system in a suitably chosen product space (see also [4]). Subsequently we will make use of the Cauchy-Lipschitz-Picard Theorem for absolutely continues trajectories (see, for example, [24, Proposition 6.2.1], [26, Theorem 54]). Notice that under $(CStrong)$ the operator $S_{t}$ in Lemma 1 is Lipschitz continuous with constant $\frac{c}{\alpha}$ for every $t\in[0,+\infty)$.

Theorem 3**.**

Assume that $(Cstrong)$ holds, and $M_{1}\in L^{1}_{loc}([0,+\infty),{\cal H})$ and $M_{2}\in L^{1}_{loc}([0,+\infty),{\cal G})$, namely,

[TABLE]

are integrable on $[0,T]$ for every $T>0$. Then, for every starting points $(x^{0},z^{0},y^{0})\in{\mathcal{H}}\times{\mathcal{G}}\times{\mathcal{G}}$, the dynamical system (4) has a unique strong global solution $(x,z,y):[0,+\infty)\longrightarrow{\mathcal{H}}\times{\mathcal{G}}\times{\mathcal{G}}.$

Proof.

Denoting $U(t)=(x(t),z(t),y(t))$, the dynamical system (4) can be rewritten as

[TABLE]

where

[TABLE]

is defined as

[TABLE]

The existence and uniqueness of a strong global solution follows according to the Cauchy-Lipschitz-Picard Theorem, if we show: (1) that $\Gamma(t,\cdot,\cdot,\cdot)$ is $L(t)$-Lipschitz continuous for every $t\in[0,+\infty)$ and the Lipschitz constant as a function of time has the property that $L(\cdot)\in L^{1}_{loc}([0,+\infty),\mathbb{R})$; (2) that $\Gamma(\cdot,x,z,y)\in L^{1}_{loc}([0,+\infty),\mathcal{H}\times\mathcal{G}\times\mathcal{G})$ for every $(x,z,y)\in\mathcal{H}\times\mathcal{G}\times\mathcal{G}$.

(1) Let $t\in[0,+\infty)$ be fixed and consider $(x,z,y),(\overline{x},\overline{z},\overline{y})\in\mathcal{H}\times\mathcal{G}\times\mathcal{G}$. We have

[TABLE]

where (see Lemma 1)

[TABLE]

Hence,

[TABLE]

Using Lemma 1 and taking into account that $(Cstrong)$ is fulfilled, which means that the Lipschitz constant of the operator $S_{t}$ is $\frac{c}{\alpha}$, it follows

[TABLE]

By taking into account the nonexpansiveness of the proximal operator and that $\gamma\in[0,1]$, it also follows

[TABLE]

Finally,

[TABLE]

Consequently,

[TABLE]

where

[TABLE]

and

[TABLE]

which means that $\Gamma(t,\cdot,\cdot,\cdot)$ is $L(t)$-Lipschitz continuous. Since $M_{1}\in L^{1}_{loc}([0,+\infty),{\cal H})$ and $M_{2}\in L^{1}_{loc}([0,+\infty),{\cal G})$, it is obvious that $L(\cdot)\in L^{1}_{loc}([0,+\infty),\mathbb{R})$.

(2) Now we will show that $\Gamma(\cdot,x,z,y)\in L^{1}_{loc}([0,+\infty),\mathcal{H}\times\mathcal{G}\times\mathcal{G})$ for every $(x,z,y)\in\mathcal{H}\times\mathcal{G}\times\mathcal{G}$. Let $(x,z,y)\in\mathcal{H}\times\mathcal{G}\times\mathcal{G}$ be fixed and $T>0$. We have

[TABLE]

By Lemma 2 and taking into account that $\alpha(t)=\alpha>0$ for every $t\in[0,+\infty)$ and $\gamma\in[0,1]$, we have for every $t\in[0,+\infty)$ that

[TABLE]

[TABLE]

and

[TABLE]

Since $M_{1}\in L^{1}_{loc}([0,+\infty),{\cal H})$ and $M_{2}\in L^{1}_{loc}([0,+\infty),{\cal G})$, it follows that the integral

[TABLE]

exists and it is finite, in other words, $\Gamma(\cdot,x,z,y)\in L^{1}_{loc}([0,+\infty),\mathcal{H}\times\mathcal{G}\times\mathcal{G})$.

Consequently, the dynamical system (17) has a unique locally absolutely continuous solution, which means that the dynamical system (14) has a unique strong global solution. ∎

3 Some technical results

In this section we will prove some technical results which will be useful in the asymptotic analysis of the dynamical system (4). We endow the real linear space $\mathcal{L}(\mathcal{H}):=\{A:\mathcal{H}\longrightarrow\mathcal{H}:A\mbox{ is linear and continuous}\}$ with the norm

[TABLE]

If $A\in\mathcal{L}(\mathcal{H})$ is self-adjoint, then it holds (see [29, Lemma 3.2.4 iv)])

[TABLE]

Definition 3**.**

We say that the map $M:[0,+\infty)\longrightarrow\mathcal{L}(\mathcal{H}),\,t\longrightarrow M(t),$ is derivable at $t_{0}\in[0,+\infty)$, if the limit

[TABLE]

taken with respect to the norm topology of $\mathcal{L}(\mathcal{H})$ exists. In this case we denote by $\dot{M}(t_{0})\in\mathcal{L}(\mathcal{H})$ the value of this limit.

If $\dot{M}(t_{0})$ exists, for $t_{0}\in[0,+\infty)$, then one can easily see that

[TABLE]

According to Remark 4, if $M$ is locally absolutely continuous then $\dot{M}(t)$ exists for almost every $t\in[0,+\infty).$

Assume now that $M(t)\in\mathcal{L}(\mathcal{H})$ is self-adjoint for every $t\in[0,+\infty)$ and that it is derivable at $t_{0}\in[0,+\infty)$. For all $x,u\in{\cal H}$ we have

[TABLE]

which shows that $\dot{M}(t_{0})$ is also self-adjoint.

Lemma 4**.**

Let $M:[0,+\infty)\longrightarrow\mathcal{L}(\mathcal{H}),\,t\longrightarrow M(t),$ be derivable at $t_{0}\in[0,+\infty)$, and let the maps $x,y:[0,+\infty)\longrightarrow\mathcal{H}$ be also derivable at $t_{0}.$ Then the real function $t\longrightarrow\langle M(t)x(t),y(t)\rangle$ is derivable at $t_{0}$ and one has

[TABLE]

Proof.

We have

[TABLE]

The derivation formula of the scalar product leads to the desired conclusion

[TABLE]

∎

The main result of this section follows.

Lemma 5**.**

Assume that $(Cstrong)$ holds and that the maps $M_{1}:[0,+\infty)\longrightarrow S_{+}(\mathcal{H})$ and $M_{2}:[0,+\infty)\longrightarrow S_{+}(\mathcal{G})$ are locally absolutely continuous. For a given starting point $(x^{0},z^{0},y^{0})\in{\mathcal{H}}\times{\mathcal{G}}\times{\mathcal{G}}$, let $(x,z,y):[0,+\infty)\longrightarrow{\mathcal{H}}\times{\mathcal{G}}\times{\mathcal{G}}$ be the unique strong global solution of the dynamical system (4). Then

[TABLE]

is locally absolutely continuous, hence $(\ddot{x}(t),\ddot{z}(t),\ddot{y}(t))$ exists for almost every $t\in[0,+\infty)$.

In addition, if $\sup_{t\geq 0}\|M_{1}(t)\|<+\infty$ and $\sup_{t\geq 0}\|M_{2}(t)\|<+\infty$, then there exists $L>0$ such that

[TABLE]

for almost every $t\in[0,+\infty).$

Proof.

Let $T>0$ be fixed. We will use the same notations as in the proof of Theorem 3. Let $t,r\in[0,T]$ be fixed. We have

[TABLE]

Since

[TABLE]

according to Lemma 1, we get

[TABLE]

Since $t\longrightarrow\|M_{1}(t)\|$ is bounded on $[0,T],$ there exists $L_{1}:=L_{1}(T)>0$ such that

[TABLE]

Similarly, since

[TABLE]

by the nonexpansiveness of the proximal operator we get

[TABLE]

Since $t\longrightarrow\|M_{2}(t)\|$ is bounded on $[0,T],$ by taking into consideration (18), one can easily see that there exists $L_{2}:=L_{2}(T)>0$ such that

[TABLE]

Further, by using (18) and (19), we get

[TABLE]

Hence, there exists $L_{3}:=c(\|A\|L_{1}+\|A\|+L_{2}+1)>0$ such that

[TABLE]

Using now Lemma 2 (i), we get

[TABLE]

Since $r\mapsto S_{r}$ and ${\nabla}h$ are Lipschitz continuous and $x,z,y$ and $M_{1}$ are absolutely continuous, the map

[TABLE]

is bounded on $[0,T].$ Consequently, there exists $L_{4}:=L_{4}(T)>0$ such that

[TABLE]

Similarly, using this time Lemma 2 (ii), we get

[TABLE]

Since the proximal operator is nonexpansive and $x,z,y$ and $M_{2}$ are absolutely continuous, the map

[TABLE]

is bounded on $[0,T].$ Consequently, there exists $L_{5}:=L_{5}(T)>0$ such that

[TABLE]

Further, by using (22) and (24), we get

[TABLE]

Consequently, there exists $L_{6}:=c(\|A\|L_{4}+L_{5})>0$ such that

[TABLE]

Summing the relations (18)-(25) we obtain that there exists $L_{7}>0$ such that

[TABLE]

Let be $\epsilon>0$. Since the maps $x,z,y,M_{1}$ and $M_{2}$ are absolutely continuous on $[0,T]$, there exists $\eta>0$ such that for any finite family of intervals $I_{k}=(a_{k},b_{k})\subseteq[0,T]$ such that for any subfamily of disjoint intervals $I_{j}$ with $\sum_{j}|b_{j}-a_{j}|<\eta$ it holds

[TABLE]

[TABLE]

Consequently,

[TABLE]

hence $\dot{U}(\cdot)=(\dot{x}(\cdot),\dot{z}(\cdot),\dot{y}(\cdot))$ is absolutely continuous on $[0,T]$. This proves that the second order derivatives $\ddot{x},\ddot{z},\ddot{y}$ exist almost everywhere on $[0,+\infty).$

We come now to the proof of the second statement and assume to this end that $\sup_{t\geq 0}\|M_{1}(t)\|<+\infty$ and $\sup_{t\geq 0}\|M_{2}(t)\|<+\infty$. Under these assumption, $L_{1},L_{2}$ and $L_{3}$ appearing in (18), (19) and (20), respectively, can be taken as being global constants, that is, (18)-(20) hold for every $t,r\in[0,+\infty)$.

Since $R_{(x(r),z(r),y(r))}(r)=\dot{x}(r)$ and $Q_{(x(r),z(r),y(r))}(r)=\dot{z}(r)$ for every $r\in[0,+\infty)$, from (21) and (23) we get

[TABLE]

and, respectively,

[TABLE]

for every $t,r\in[0,+\infty).$ Consequently,

[TABLE]

for every $t,r\in[0,+\infty).$ This shows that there exists $L>0$ such that

[TABLE]

for every $t,r\in[0,+\infty).$

Now we fix $r\in[0,+\infty)$ at which the second derivative of the trajectories exist and take in the above inequality $t=r+h$ for some $h>0$. This yields

[TABLE]

After dividing in the above inequality by $h$ and letting $h\longrightarrow 0$, we obtain

[TABLE]

This inequality holds for almost every $r\in[0,+\infty).$ ∎

4 Asymptotic analysis

In this section we will address the asymptotic behaviour of the trajectories generated by the dynamical system (4). At the beginning we will recall two results which will play a central role in the asymptotic analysis (see [2, Lemma 5.1] and [2, Lemma 5.2], respectively).

Lemma 6**.**

Suppose that $A:[0,+\infty)\rightarrow\mathbb{R}$ is locally absolutely continuous and bounded from below and that there exists $B\in L^{1}([0,+\infty),\mathbb{R})$ such that for almost every $t\in[0,+\infty)$

[TABLE]

Then there exists $\lim_{t\rightarrow+\infty}A(t)\in\mathbb{R}$.

Lemma 7**.**

If $1\leq p<\infty$, $1\leq r\leq\infty$, $A:[0,+\infty)\rightarrow[0,+\infty)$ is locally absolutely continuous, $A\in L^{p}([0,+\infty),\mathbb{R})$, $B:[0,+\infty)\rightarrow\mathbb{R}$, $B\in L^{r}([0,+\infty),\mathbb{R})$ and for almost every $t\in[0,+\infty)$

[TABLE]

then $\lim_{t\rightarrow+\infty}A(t)=0$.

The first result which we prove in this section is a continuous version of the Opial Lemma formulated in the setting of variable metrics (see [18, Theorem 3.3] for its discrete counterpart).

Lemma 8**.**

Let $\mathcal{C}\subseteq{{\mathcal{H}}}$ be a nonempty set and $x:[0,+\infty)\rightarrow{{\mathcal{H}}}$ a continuous map. Let $M:[0,+\infty)\longrightarrow S_{+}(\mathcal{H})$ be such that $M(t_{1})\succcurlyeq M(t_{2})$ for every $t_{1},t_{2}\in[0,+\infty)$ with $t_{1}\leq t_{2}$ and there exists $\alpha>0$ with $M(t)\in P_{\alpha}(\mathcal{H})$ for every $t\in[0,+\infty)$. If the following two conditions are fulliled

(i) the limit $\lim_{t\rightarrow+\infty}\|x(t)-z\|_{M(t)}$ exists for every $z\in\mathcal{C}$;

(ii) every weak sequential cluster point of $x(t),t\in[0,+\infty),$ belongs to $\mathcal{C}$;

then there exists $x_{\infty}\in\mathcal{C}$ such that $x(t),t\in[0,+\infty)$, converges weakly to $x_{\infty}$ as $t\rightarrow+\infty$.

Proof.

Since $\mathcal{C}\neq\emptyset$ and $M(t)\in P_{\alpha}(\mathcal{H})$, by (i) we have that $x$ is bounded, hence it possesses at least one weak sequential cluster point, which belongs to ${\cal C}$. We show that $x$ has exactly one weak sequential cluster point.

Indeed, let $x_{1},x_{2}$ two weak sequential cluster points of $x$. For our claim it is enough to show that $x_{1}=x_{2}$. Obviously $x_{1},x_{2}\in\mathcal{C}$ and there exist the sequences $(t_{n}^{1})_{n\geq 0},(t_{n}^{2})_{n\geq 0}\subseteq[0,+\infty)$ with $\lim_{n\longrightarrow+\infty}t_{n}^{1}=+\infty$ and $\lim_{n\longrightarrow+\infty}t_{n}^{2}=+\infty$ such that $(x(t_{n}^{1}))_{n\geq 0}$ converges weakly to $x_{1}$ and $(x(t_{n}^{2}))_{n\geq 0}$ converges weakly to $x_{2}$ as $n\rightarrow+\infty$.

Further, since $M(t_{1})\succcurlyeq M(t_{2})$ for every for every $t_{1},t_{2}\in[0,+\infty)$ with $t_{1}\leq t_{2}$ and $M(t)\in P_{\alpha}(\mathcal{H})$ for every $t\in[0,+\infty),$ it follows that for every $z\in\mathcal{H}$ the function

[TABLE]

is decreasing and is bounded from below, hence there exists

[TABLE]

Since $x_{1},x_{2}\in\mathcal{C}$, we have that the limits $\lim_{t\rightarrow+\infty}\|x(t)-x_{1}\|_{M(t)}^{2}$ and $\lim_{t\rightarrow+\infty}\|x(t)-x_{2}\|_{M(t)}^{2}$ exist. Further, since

[TABLE]

holds for every $t\in[0,+\infty)$, the limit

[TABLE]

exists.

Next we show that the limits

[TABLE]

exists for every $z\in{\cal H}$. To this end we fix $z\in{\cal H}$. We will actually show that

[TABLE]

and the conclusion will follow by the Cauchy criterion.

For $U\in S_{+}(\mathcal{H})$ we have by the generalized Cauchy-Schwarz inequality that

[TABLE]

Hence, for $t,s\in[0,+\infty)$ with $t\leq s$ we have $M(t)-M(s)\in S_{+}(\mathcal{H})$, therefore

[TABLE]

Since $M(0)\succcurlyeq M(t)$, we have that

[TABLE]

for every $t\in[0,+\infty)$. This shows that $\|M(t)-M(s)\|,t,s\in[0,+\infty),$ is bounded. This, together with the fact that $\lim\limits_{s,t\rightarrow+\infty}\|z\|_{(M(t)-M(s))}^{2}=0$, which follows from (26), implies

[TABLE]

This proves (28). For every $z\in{\cal H}$ let us denote by $Mz:=\lim_{t\longrightarrow+\infty}M(t)z.$ Since $M(t)\in P_{\alpha}(\mathcal{H})$ for every $t\in[0,+\infty)$, it holds

[TABLE]

Since $(x(t_{n}^{1}))_{n\geq 0}$ converges weakly to $x_{1}$ and $(x(t_{n}^{2}))_{n\geq 0}$ converges weakly to $x_{2}$ as $n\rightarrow+\infty$ and

[TABLE]

passing to the limit in (27) we get

[TABLE]

and

[TABLE]

In conclusion,

[TABLE]

which shows that $x_{1}=x_{2}.$ ∎

Remark 5**.**

If a map $M:[0,+\infty)\longrightarrow S_{+}(\mathcal{H})$ satisfies $M(t_{1})\succcurlyeq M(t_{2})$ for every $t_{1},t_{2}\in[0,+\infty)$ with $t_{1}\leq t_{2}$ we say that $M$ is monotonically decreasing. If $M$ is monotonically decreasing and locally absolutely continuous, then $\dot{M}(t)$ exists and $\langle\dot{M}(t)x,x\rangle\leq 0$ for almost every $t\in[0,+\infty).$

The following result is an adaptation of a result from [3] to our setting.

Proposition 9**.**

(see [3, Proposition 2.4]) In the setting of the optimization problem (1), let $(a_{n},a_{n}^{*})_{n\geq 0}$ be a sequence in the graph of ${\partial}(f+h)$ and $(b_{n},b_{n}^{*})_{n\geq 0}$ a sequence in the graph of ${\partial}g.$ Suppose that $a_{n}$ converges weakly to $\overline{x}\in\mathcal{H},$ $b_{n}^{*}$ converges weakly to $\overline{v}\in\mathcal{G}$, $a_{n}^{*}+A^{*}b_{n}^{*}\longrightarrow 0,$ and $Aa_{n}-b_{n}\longrightarrow 0$ as $n\rightarrow+\infty$. Then

[TABLE]

and

[TABLE]

The theorem which states the asymptotic convergence of the trajectories generated by the dynamical system (4) to a saddle point of the Lagrangian of the problem (1) follows.

Theorem 10**.**

In the setting of the optimization problem (1), assume that the set of saddle points of the Lagrangian $l$ is nonempty, the maps

[TABLE]

are locally absolutely continuous and monotonically decreasing,

[TABLE]

and

[TABLE]

For an arbitrary starting point $(x^{0},z^{0},y^{0})\in{\mathcal{H}}\times{\mathcal{G}}\times{\mathcal{G}}$, let $(x,z,y):[0,+\infty)\longrightarrow{\mathcal{H}}\times{\mathcal{G}}\times{\mathcal{G}}$ be the unique strong global solution of the dynamical system (4). If one of the following conditions holds:

• (I)

there exists $\alpha>0$ such that $M_{1}(t)+\frac{c(1-\gamma)}{4}A^{*}A-\frac{L_{h}}{4}I\in P_{\alpha}(\mathcal{H})$ for every $t\in[0,+\infty)$;

• (II)

$\gamma\in[0,1)$* and there exists $\alpha>0$ such that $A^{*}A\in P_{\alpha}(\mathcal{H})$;*

then the trajectory $(x(t),z(t),y(t))$ converges weakly to a saddle point of $l$ as $t\longrightarrow+\infty.$

Proof.

The proof of the theorem relies on Lemma 8. An important step in the proof will be the derivation of two inequalities of Lyapunov type, namely, (4), in the case when $L_{h}\neq 0$, and (4), in the case when $L_{h}=0$. Let $(x^{*},z^{*},y^{*})\in{\mathcal{H}}\times{\mathcal{G}}\times{\mathcal{G}}$ be a saddle point of the Lagrangian $l$. Then

[TABLE]

According to (5) we have for almost every $t\in[0,+\infty)$

[TABLE]

which yields, by taking into account the monotonicity of $\partial f$,

[TABLE]

Similarly, according to (6) we have for almost every $t\in[0,+\infty)$

[TABLE]

which yields, by taking into account the monotonicity of $\partial g$,

[TABLE]

By using the last equation of (4) we obtain for almost every $t\in[0,+\infty)$

[TABLE]

Assume that $L_{h}>0$. By using the Baillon-Haddad Theorem we have for almost every $t\in[0,+\infty)$

[TABLE]

By summing (30) and (32) and by taking into account (4) and (4) we obtain for almost every $t\in[0,+\infty)$

[TABLE]

We have for almost every $t\in[0,+\infty)$

[TABLE]

Since

[TABLE]

and

[TABLE]

we obtain from above that for almost every $t\in[0,+\infty)$ it holds

[TABLE]

By using Lemma 4 we observe that for almost every $t\in[0,+\infty)$ it holds

[TABLE]

and

[TABLE]

By plugging the last two identities and (4) into (4), we obtain for almost every $t\in[0,+\infty)$

[TABLE]

According to Remark 5,

[TABLE]

for almost every $t\in[0,+\infty)$. This means that for almost every $t\in[0,+\infty)$ we have

[TABLE]

From Lemma 6 we have

[TABLE]

Let be $T>0$. By integrating (4) on the interval $[0,T]$ we obtain

[TABLE]

Letting $T$ converge to $+\infty$ we find

[TABLE]

and, consequently,

[TABLE]

In the case when $L_{h}=0$, which corresponds to the situation when $h$ is an affine-continuous function, instead of (4) we obtain that for almost every $t\in[0,+\infty)$

[TABLE]

By arguing as above, we obtain also in this case (38), (39)-(41) and (42).

Further, we have that $\dot{x}(\cdot)\in L^{2}([0,+\infty),{\mathcal{H}})$. Indeed, in case (I), when we assume that there exists $\alpha>0$ such that $M_{1}(t)+\frac{(1-\gamma)c}{4}A^{*}A-\frac{L_{h}}{4}I\in P_{\alpha}(\mathcal{H})$ for every $t\in[0,+\infty)$, then this yields automatically. In case (II), from $(1-\gamma)A\dot{x}(\cdot)\in L^{2}([0,+\infty),{\mathcal{G}})$ and $\gamma\in[0,1)$, we have

[TABLE]

But, since $A^{*}A\in P_{\alpha}(H)$, it yields $\|A\dot{x}(t)\|^{2}\geq\alpha\|\dot{x}(t)\|^{2}$ for almost every $t\in[0,+\infty)$, which means that also in this case

[TABLE]

According to Lemma 5, this yields

[TABLE]

Consequently, for almost every $t\in[0,+\infty)$ it holds

[TABLE]

and the right-hand side is a function in $L^{1}([0,+\infty),\mathbb{R})$. Hence, according to Lemma 7,

[TABLE]

Similarly, we obtain that

[TABLE]

We will close the proof of the theorem by showing that the asymptotic convergence of the trajectory follows from Lemma 8. One can easily notice that (38) is nothing else but condition (i) of this lemma when applied in the product space for the trajectory

[TABLE]

the monotonically decreasing map

[TABLE]

and the set $\mathcal{C}$ taken as the set of saddle points of the Lagrangian $l.$

Next we will show that also condition (ii) in Lemma 8 is fulfilled, namely, that every weak sequential cluster point of the trajectory $(x(t),z(t),y(t)),t\in[0,+\infty),$ is a saddle point of the Langrangian $l$.

Let $(\overline{x},\overline{z},\overline{y})$ be such a weak sequentially cluster point. This means that there exists a sequence $(s_{n})_{n\geq 0}$ with $s_{n}\longrightarrow+\infty$ such that $(x(s_{n}),z(s_{n}),y(s_{n}))$ converges to $(\overline{x},\overline{z},\overline{y})$ as $n\longrightarrow+\infty$ in the weak topology of $\mathcal{H}\times\mathcal{G}\times\mathcal{G}$.

From (29) and (31) we get for every $n\geq 0$

[TABLE]

and

[TABLE]

respectively. For every $n\geq 0$, let

[TABLE]

and

[TABLE]

Hence, $(a_{n},a_{n}^{*})_{n\geq 0}\subseteq\operatorname*{Gr}{\partial}(f+h).$ Similarly, for every $n\geq 0$, let

[TABLE]

and

[TABLE]

Hence, $(b_{n},b_{n}^{*})_{n\geq 0}\subseteq\operatorname*{Gr}{\partial}g.$

Since $\lim_{t\longrightarrow+\infty}\dot{x}(t)=0$, $\lim_{t\longrightarrow+\infty}\dot{z}(t)=0$ and $\lim_{t\longrightarrow+\infty}\dot{y}(t)=0$ it follows that $(a_{n})_{n\geq 0}$ converges weakly to $\overline{x}$ as $n\rightarrow\infty$. Furthermore, since $(M_{2}(s_{n}))_{n\geq 0}$ is bounded, and

[TABLE]

it follows that $(b_{n}^{*})_{n\geq 0}$ converges weakly to $\overline{y}$ as $n\rightarrow\infty$.

From (14) we have

[TABLE]

which implies that $A\overline{x}=\overline{z}$. On the other hand, since ${\nabla}h$ is Lipschitz continuous, we have

[TABLE]

hence

[TABLE]

Thus, according to Proposition 9, we have

[TABLE]

Consequently, $(\overline{x},\overline{z},\overline{y})$ is a saddle point of $l.$

The conclusion of the theorem follows from Lemma 8. ∎

Next we will address two particular cases of the dynamical system (4). We consider first the case when $M_{1}(t)=M_{2}(t)=0$ for every $t\in[0,+\infty)$, thus, the system (4) becomes

[TABLE]

where $c>0$ and $\gamma\in[0,1]$. The dynamical system (44) can be seen as the continuous counterpart of the classical ADMM algorithm. The corresponding convergence result follows as a particular case of Theorem 10.

Theorem 11**.**

In the setting of the optimization problem (1), assume that the set of saddle points of the Lagrangian $l$ is nonempty, $\gamma\in[0,1)$ and that there exists $\alpha>0$ such that $A^{*}A-\frac{L_{h}}{c(1-\gamma)}I\in P_{\alpha}(\mathcal{H})$. For an arbitrary starting point $(x^{0},z^{0},y^{0})\in{\mathcal{H}}\times{\mathcal{G}}\times{\mathcal{G}}$, let $(x,z,y):[0,+\infty)\longrightarrow{\mathcal{H}}\times{\mathcal{G}}\times{\mathcal{G}}$ be the unique strong global solution of the dynamical system (44). Then the trajectory $(x(t),z(t),y(t))$ converges weakly to a saddle point of $l$ as $t\longrightarrow+\infty.$

Next we consider the setting from Remark 2 with $M_{1}(t)=\frac{1}{\tau(t)}I-cA^{*}A$ and $M_{2}(t)=0$, where $\tau(t)$ is such that $c\tau(t)\|A\|^{2}\leq 1$, for every $t\in[0,+\infty)$. The resulting dynamical system is the primal-dual system (8). The corresponding convergence result follows again as a particular case of Theorem 10.

Theorem 12**.**

In the setting of the optimization problem (1), assume that the set of saddle points of the Lagrangian $l$ is nonempty, the map $\tau:[0,+\infty)\rightarrow(0,+\infty)$ is locally absolutely continuous and monotonically increasing with

[TABLE]

and $\sup_{t\geq 0}\frac{\tau^{\prime}(t)}{\tau^{2}(t)}<+\infty$. For an arbitrary starting point $(x^{0},z^{0},y^{0})\in{\mathcal{H}}\times{\mathcal{G}}\times{\mathcal{G}}$, let $(x,z,y):[0,+\infty)\longrightarrow{\mathcal{H}}\times{\mathcal{G}}\times{\mathcal{G}}$ be the unique strong global solution of the dynamical system (8). If one of the following assumptions holds:

• (I)

there exists $\alpha>0$ such that $\frac{4-\tau(t)L_{h}}{4\tau(t)}I-\frac{c(3+\gamma)}{4}A^{*}A\in P_{\alpha}(\mathcal{H})$ for every $t\in[0,+\infty)$;

• (II)

$\gamma\in[0,1)$* and there exists $\alpha>0$ such that $A^{*}A\in P_{\alpha}(\mathcal{H})$;*

then the trajectory $(x(t),z(t),y(t))$ converges weakly to a saddle point of $l$ as $t\longrightarrow+\infty.$

Remark 6**.**

Let be $t\in[0,+\infty)$. Notice that the condition $\frac{4-\tau(t)L_{h}}{4\tau(t)}I-\frac{c(3+\gamma)}{4}A^{*}A\in S_{+}(\mathcal{H})$ is fulfilled if and only if

[TABLE]

On the other hand, the condition $\frac{4-\tau(t)L_{h}}{4\tau(t)}I-\frac{c(3+\gamma)}{4}A^{*}A\in P_{\alpha}(\mathcal{H})$ holds, for $\alpha>0$, if and only if

[TABLE]

For the last result of this paper we go back to the general dynamical system (4) and provide convergence rates for the violation of the feasibility condition by ergodic trajectories and the convergence of the objective function along these ergodic trajectories to its minimal value. The result can be seen as the continuous counterpart of a convergence rate result proved for the ADMM algorithm in [22, Theorem 4.3].

Theorem 13**.**

In the setting of the optimization problem (1), assume that the set of saddle points of the Lagrangian $l$ is nonempty, the maps

[TABLE]

are locally absolutely continuous and monotonically decreasing,

[TABLE]

[TABLE]

and that one of the following conditions holds:

• (I)

there exists $\alpha>0$ such that $M_{1}(t)+\frac{c(1-\gamma)}{4}A^{*}A-\frac{L_{h}}{4}I\in P_{\alpha}(\mathcal{H})$ for every $t\in[0,+\infty)$;

• (II)

$\gamma\in[0,1)$* and there exists $\alpha>0$ such that $A^{*}A\in P_{\alpha}(\mathcal{H})$;*

For an arbitrary starting point $(x^{0},z^{0},y^{0})\in{\mathcal{H}}\times{\mathcal{G}}\times{\mathcal{G}}$, let $(x,z,y):[0,+\infty)\longrightarrow{\mathcal{H}}\times{\mathcal{G}}\times{\mathcal{G}}$ be the unique strong global solution of the dynamical system (4). Consider further for every $t\in(0,+\infty)$ the ergodic trajectories

[TABLE]

and

[TABLE]

Then there exists $K\geq 0$ such that for every $t\in(0,+\infty)$

[TABLE]

In addition, for every $\overline{x}\in\mathcal{H}$ and every $t\in(0,+\infty)$ such that $(\tilde{x}(t),\tilde{z}(t))\in\operatorname*{dom}f\times\operatorname*{dom}g$, one has

[TABLE]

where

[TABLE]

Proof.

Let $\overline{x}\in\mathcal{H}$ be fixed. By using (5), that is

[TABLE]

it yields

[TABLE]

for almost every $t\in[0,+\infty)$. Similarly, by using (31), that is

[TABLE]

it yields

[TABLE]

for almost every $t\in[0,+\infty)$. Further, by using the convexity of $h$ and the Descent Lemma we obtain for almost every $t\in[0,+\infty)$

[TABLE]

Adding (45) and (4) we obtain for almost every $t\in[0,+\infty)$

[TABLE]

We recall the following four identities from the proof of Theorem 10 (here were actually replace $x^{*}$ with $\overline{x}$ and $y^{*}$ by [math])

[TABLE]

which corresponds to (4),

[TABLE]

which corresponds to (4), and

[TABLE]

and

[TABLE]

which all hold for for almost every $t\in[0,+\infty)$. By adding the four identities, (48) and (46), we obtain for almost every $t\in[0,+\infty)$

[TABLE]

By neglecting the negative terms (here we use also that $M_{1}(t)+\frac{c(1-\gamma)}{4}A^{*}A-\frac{L_{h}}{2}I\in S_{+}(\mathcal{H})$), we obtain for almost every $t\in[0,+\infty)$

[TABLE]

where

[TABLE]

For $\tilde{x}(t)=\frac{1}{t}\int_{0}^{t}(\dot{x}(s)+x(s))ds$ and $\tilde{z}(t)=\frac{1}{t}\int_{0}^{t}(\dot{z}(s)+z(s))ds,$ it holds

[TABLE]

From Theorem 10 it follows that the trajectory $(x(t),z(t),y(t))\to(x_{\infty},z_{\infty},y_{\infty}),t\in[0,+\infty)$, converges weakly to a saddle point of $l$ as $t\rightarrow+\infty$. This means that $y(t),t\in[0,+\infty),$ it is bounded, thus there exists $K\geq 0$ such that

[TABLE]

Let $t\in(0,+\infty)$ be such that $(\tilde{x}(t),\tilde{z}(t))\in\operatorname*{dom}f\times\operatorname*{dom}g$. By Jensen’s inequality in the integral form we have for every $t\in(0,+\infty)$

[TABLE]

and

[TABLE]

which, combined with (4), yields

[TABLE]

Hence,

[TABLE]

∎