Generalized Conditional Gradient with Augmented Lagrangian for Composite Minimization

TL;DR

This paper introduces CGALP, a hybrid optimization algorithm combining generalized conditional gradient and augmented Lagrangian methods, capable of efficiently solving complex convex composite problems with affine constraints.

Contribution

The paper proposes a novel splitting scheme, CGALP, that relaxes Lipschitz gradient requirements and handles multiple convex functions with affine constraints using an augmented Lagrangian approach.

Findings

Proves asymptotic feasibility and boundedness of dual multipliers.

Establishes convergence of Lagrangian values to saddle-point optimality.

Provides convergence rates for feasibility gap and Lagrangian values.

Abstract

In this paper we propose a splitting scheme which hybridizes generalized conditional gradient with a proximal step which we call CGALP algorithm, for minimizing the sum of three proper convex and lower-semicontinuous functions in real Hilbert spaces. The minimization is subject to an affine constraint, that allows in particular to deal with composite problems (sum of more than three functions) in a separate way by the usual product space technique. While classical conditional gradient methods require Lipschitz-continuity of the gradient of the differentiable part of the objective, CGALP needs only differentiability (on an appropriate subset), hence circumventing the intricate question of Lipschitz continuity of gradients. For the two remaining functions in the objective, we do not require any additional regularity assumption. The second function, possibly nonsmooth, is assumed simple, i.e., the associated proximal mapping is easily computable. For the third function, again nonsmooth, we just assume that its domain is also bounded and that a linearly perturbed minimization oracle is accessible. In particular, this last function can be chosen to be the indicator of a nonempty bounded closed convex set, in order to deal with additional constraints. Finally, the affine constraint is addressed by the augmented Lagrangian approach. Our analysis is carried out for a wide choice of algorithm parameters satisfying so called "open loop" rules. As main results, under mild conditions, we show asymptotic feasibility with respect to the affine constraint, boundedness of the dual multipliers, and convergence of the Lagrangian values to the saddle-point optimal value. We also provide (subsequential) rates of convergence for both the feasibility gap and the Lagrangian values.

Equations

$$\underset{x\in {\mathcal{H}}_p}{\min }\left\{f(x)+g(Tx)+h(x):Ax=b\right\},$$

$$\begin{array}{r}\underset{x\in {\mathcal{H}}_p}{\min }\left\{f(x):x\in \mathcal{C}\right\}\end{array}$$

$$\underset{s\in \mathcal{H}}{\min }h\left(s\right)+⟨x,s⟩$$

$$\begin{array}{r}\underset{x\in \mathcal{C}}{\min }\left\{f(x)+g\left(Tx\right)\right\}.\end{array}$$

$$\begin{array}{r}\underset{x\in \mathcal{C}}{\min }\left\{f(x):Ax=0\right\}.\end{array}$$

$$\begin{array}{r}{g}^{\ast }\left(u\right)\stackrel{\text{def}}{=}\underset{x\in \mathcal{H}}{\sup }\left\{⟨u,x⟩-g\left(x\right)\right\}.\end{array}$$

$$g_1\le g_2⟹g_2^{\ast }\le g_1^{\ast }.$$

$$\begin{array}{r}{\mathrm{prox}}_g\left(x\right)\stackrel{\text{def}}{=}\underset{y\in \mathcal{H}}{\mathrm{argmin}}\left\{g(y)+\frac{1}{2}{\Vert x-y\Vert }^2\right\}\end{array}$$

$$g^{\beta }\left(x\right)\stackrel{\text{def}}{=}\underset{y\in \mathcal{H}}{\inf }\left\{g(y)+\frac{1}{2\beta }{\Vert x-y\Vert }^2\right\}.$$

$$2\left[g(x^{+})-g(y)\right]+\Vert x^{+}-y{\Vert }^2-{\Vert x-y\Vert }^2+\Vert x^{+}-x{\Vert }^2\le 0.$$

$$\partial g(x)=\big{\{}u\in\mathcal{H}:\;g(y)\geq g(x)+\left\langle u,y-x\right\rangle\quad\forall y\in\mathcal{H}\big{\}}.$$

$$\{\begin{array}{ll}\frac{\partial }{\partial \beta }{g}^{\beta }\left(x\right)=-\frac{1}{2}{\Vert {\nabla }_x{g}^{\beta }\left(x\right)\Vert }^2& \left(x,\beta \right)\in \mathcal{H}\times (0,+\infty )\\ g^0\left(x\right)=g\left(x\right)& x\in \mathcal{H}.\end{array}$$

$$\begin{array}{r}{\nabla }_x{g}^{\beta }\left(x\right)=\frac{x-{\mathrm{prox}}_{\beta g}\left(x\right)}{\beta }.\end{array}$$

$$\begin{array}{rl}0\le g^{{\beta }^{\prime }}\left(x\right)-g^{\beta }\left(x\right)& \le \frac{\beta -{\beta }^{\prime }}{2}{\Vert {\nabla }_x{g}^{{\beta }^{\prime }}\left(x\right)\Vert }^2;\\ 0\le g\left(x\right)-g^{\beta }\left(x\right)& \le \frac{\beta }{2}{\Vert {\left[\partial g\left(x\right)\right]}^0\Vert }^2.\end{array}$$

$$\{\begin{array}{ll}\frac{\partial }{\partial \beta }g\left(x,\beta \right)+H\left({\nabla }_{x}g\left(x,\beta \right)\right)=0& \left(x,\beta \right)\in \mathcal{H}\times (0,+\infty )\\ g\left(x,0\right)=g\left(x\right)& x\in \mathcal{H}.\end{array}$$

$$g\left(x,t\right)\stackrel{\text{def}}{=}\underset{y\in \mathcal{H}}{\inf }\left\{g_0(y)+t{H}^{\ast }\left(\frac{y-x}{t}\right)\right\}.$$

$$\xi \left(s\right)\stackrel{\text{def}}{=}{\int }_0^1\omega \left(st\right)dt$$

$$\Vert \nabla g\left(x\right)-\nabla g\left(y\right)\Vert \le \omega \left(\Vert x-y\Vert \right).$$

$$g\left(y\right)-g\left(x\right)\le ⟨\nabla g\left(x\right),y-x⟩+\Vert y-x\Vert \xi \left(\Vert y-x\Vert \right),$$

$$\begin{array}{rl}g\left(y\right)-g\left(x\right)& ={\int }_0^1\frac{d}{dt}g\left(x+t\left(y-x\right)\right)dt\\ & ={\int }_0^1⟨\nabla g\left(x\right),y-x⟩dt+{\int }_0^1⟨\nabla g\left(x+t\left(y-x\right)\right)-\nabla g\left(x\right),y-x⟩dt\\ & \le ⟨\nabla g\left(x\right),y-x⟩+\Vert y-x\Vert {\int }_0^1\Vert \nabla g\left(x+t\left(y-x\right)\right)-\nabla g\left(x\right)\Vert dt\\ & \le ⟨\nabla g\left(x\right),y-x⟩+\Vert y-x\Vert {\int }_0^1\omega \left(t\Vert y-x\Vert \right)dt,\end{array}$$

$$\begin{array}{r}g\left(y\right)-g\left(x\right)\le ⟨\nabla g\left(x\right),y-x⟩+\frac{L}{1+\nu }{\Vert y-x\Vert }^{1+\nu },\end{array}$$

$$D_G(x,y)=G(x)-G(y)-⟨\nabla G(y),x-y⟩.$$

$$g(y)\le g(x)+⟨\nabla g(x),y-x⟩+D_G(y,x).$$

$$\left(G-g\right)(y)\ge \left(G-g\right)(x)+⟨\nabla \left(G-g\right)(x),y-x⟩.$$

$$K_{\left(G,\zeta ,\mathcal{C}\right)}\stackrel{\text{def}}{=}\underset{\begin{array}{c}x,s\in \mathcal{C};\gamma \in ]0,1]\\ z=x+\gamma \left(s-x\right)\end{array}}{\sup }\frac{D_G(z,x)}{\zeta \left(\gamma \right)}<+\infty .$$

$$\begin{array}{rl}g\left(x+\gamma \left(s-x\right)\right)& \le g(x)+\gamma ⟨\nabla g(x),s-x⟩+D_G(x+\gamma \left(s-x\right),x)\\ & \le g(x)+\gamma ⟨\nabla g(x),s-x⟩+K_{\left(G,\zeta ,\mathcal{C}\right)}\zeta \left(\gamma \right).\end{array}$$

$$\begin{array}{rl}{K}_{\left(g,\zeta ,\mathcal{C}\right)}& =\underset{\begin{array}{c}x,s\in \mathcal{C};\gamma \in ]0,1]\\ z=x+\gamma \left(s-x\right)\end{array}}{\sup }\frac{D_g(z,x)}{\zeta \left(\gamma \right)}\\ & =\underset{\begin{array}{c}x,s\in \mathcal{C};\gamma \in ]0,1]\\ z=x+\gamma \left(s-x\right)\end{array}}{\sup }\frac{g(z)-g(x)-⟨\nabla g(x),z-x⟩}{\xi (d_{\mathcal{C}}\gamma )}\\ & \le \underset{\begin{array}{c}x,s\in \mathcal{C};\gamma \in ]0,1]\\ z=x+\gamma \left(s-x\right)\end{array}}{\sup }\frac{\Vert z-x\Vert \xi \left(\Vert z-x\Vert \right)}{\xi (d_{\mathcal{C}}\gamma )}\\ & =\underset{\begin{array}{c}x,s\in \mathcal{C};\gamma \in ]0,1]\end{array}}{\sup }\frac{\gamma \Vert s-x\Vert \xi \left(\gamma \Vert s-x\Vert \right)}{\xi (d_{\mathcal{C}}\gamma )}\\ & \le \underset{\gamma \in ]0,1]}{\sup }\frac{\gamma d_{\mathcal{C}}\xi \left(d_{\mathcal{C}}\gamma \right)}{\xi (d_{\mathcal{C}}\gamma )}=d_{\mathcal{C}}.\end{array}$$

$$\begin{array}{r}{\sigma }_{\mathcal{C}}(x)=0⟺x\in (\mathrm{par}(\mathcal{C}){)}^{\perp }.\end{array}$$

$$\begin{array}{r}{g}^{d,\infty }\left(x\right)\stackrel{\text{def}}{=}\underset{\alpha \to \infty }{\lim }\frac{g\left(d+\alpha x\right)-g\left(d\right)}{\alpha }.\end{array}$$

$$r_{k+1}\le r_k-a_k+z_k,\forall k\in \mathbb{N}.$$

Full text

Key words. Conditional gradient; Augmented Lagrangian; Composite minimization; Proximal mapping; Moreau envelope.

AMS subject classifications. 49J52, 65K05, 65K10.

1 Introduction

1.1 Problem Statement

In this work, we consider the composite optimization problem,

[TABLE]

where $\mathcal{H}_{p},\mathcal{H}_{d},\mathcal{H}_{v}$ are real Hilbert spaces (the subindices $p,d$ and $v$ denoting the “primal”, the “dual”  and an auxiliary space - respectively), endowed with the associated scalar products and norms (to be understood from the context), $A:\mathcal{H}_{p}\to\mathcal{H}_{d}$ and $T:\mathcal{H}_{p}\to\mathcal{H}_{v}$ are bounded linear operators, $b\in\mathcal{H}_{d}$ and $f$, $g$, $h$ are proper, convex, and lower semi-continuous functions with $\mathcal{C}\stackrel{{\scriptstyle\text{\rm\tiny def}}}{{=}}\mathrm{dom}\left(h\right)$ being a weakly compact subset of $\mathcal{H}_{p}$. We allow for some asymmetry in regularity between the functions involved in the objective. While $g$ is assumed to be prox-friendly, for $h$ we assume that it is easy to compute a linearly-perturbed oracle (see (1.2)). On the other hand, $f$ is assumed to be differentiable and satisfies a condition that generalizes Lipschitz-continuity of the gradient (see Definition 2.6).

Problem ($\mathrsfs{P}$) can be seen as a generalization of the classical Frank-Wolfe problem in [15] of minimizing a Lipschitz-smooth function $f$ on a convex closed bounded subset $\mathcal{C}\subset\mathcal{H}_{p}$,

[TABLE]

In fact, if $A\equiv 0$, $b\equiv 0$, $g\equiv 0$, and $h\equiv\iota_{\mathcal{C}}$ is the indicator function of $\mathcal{C}$ then we recover exactly (1.1) from ($\mathrsfs{P}$).

1.2 Contribution

We develop and analyze a novel algorithm to solve ($\mathrsfs{P}$) which combines penalization for the nonsmooth function $g$ with the augmented Lagrangian method for the affine constraint $Ax=b$. In turn, this achieves full splitting of all the parts in the composite problem ($\mathrsfs{P}$) by using the proximal mapping of $g$ (assumed prox-friendly) and a linear oracle for $h$ of the form (1.2). Our analysis shows that the sequence of iterates is asymptotically feasible for the affine constraint, that the sequence of dual variables converges weakly to a solution of the dual problem, that the associated Lagrangian converges to optimality, and establishes convergence rates for a family of sequences of step sizes and sequences of smoothing/penalization parameters which satisfy so-called "open loop" rules in the sense of [31] and [13]. This means that the allowable sequences of parameters do not depend on the iterates, in contrast to a "closed loop" rule, e.g. line search or other adaptive step sizes. Our analysis also shows, in the case where ($\mathrsfs{P}$) admits a unique minimizer, weak convergence of the whole sequence of primal iterates to the solution.

The structure of ($\mathrsfs{P}$) generalizes (1.1) in several ways. First, we allow for a possibly nonsmooth term $g$. Second, we consider $h$ beyond the case of an indicator function where the linear oracle of the form

[TABLE]

can be easily solved. Observe that (1.2) has a solution over $\mathrm{dom}(h)$ since the latter is weakly compact. This oracle is reminiscent of that in the generalized conditional gradient method [7, 8, 5, 3]. Third, the regularity assumptions on $f$ are also greatly weakened to go far beyond the standard Lipschitz gradient case. Finally, handling an affine constraint in our problem means that our framework can be applied to the splitting of a wide range of composite optimization problems, through a product space technique, including those involving finitely many functions $h_{i}$ and $g_{i}$, and, in particular, intersection of finitely many nonempty bounded closed convex sets; see Section 5.

1.3 Relation to prior work

In the 1950’s Frank and Wolfe developed the so-called Frank-Wolfe algorithm in [15], also commonly referred to as the conditional gradient algorithm [24, 12, 13], for solving problems of the form (1.1). The main idea is to replace the objective function $f$ with a linear model at each iteration and solve the resulting linear optimization problem; the solution to the linear model is used as a step direction and the next iterate is computed as a convex combination of the current iterate and the step direction. We generalize this setting to include composite optimization problems involving both smooth and nonsmooth terms, intersection of multiple constraint sets, and also affine constraints.

Frank-Wolfe algorithms have received a lot of attention in the modern era due to their effectiveness in fields with high-dimensional problems like machine learning and signal processing (without being exhaustive, see, e.g., [20, 6, 22, 17, 39, 26, 10]). In the past, composite, constrained problems like ($\mathrsfs{P}$) have been approached using proximal splitting methods, e.g. generalized forward-backward as developed in [32] or forward-douglas-rachford [25]. Such approaches require one to compute the proximal mapping associated to the function $h$. Alternatively, when the objective function satisfies some regularity conditions and when the constraint set is well behaved, one can forgo computing a proximal mapping, instead computing a linear minimization oracle. The computation of the proximal step can be prohibitively expensive; for example, when $h$ is the indicator function of the nuclear norm ball, computing the proximal operator of $h$ requires a full singular value decomposition while the linear minimization oracle over the nuclear norm ball requires only the leading singular vector to be computed ([21], [38]). Unfortunately, the regularity assumptions required by classical Frank-Wolfe style algorithms are too restrictive to apply to general problems like ($\mathrsfs{P}$).

While finalizing this work, we became aware of the recent work of [37], who independently developed a conditional gradient-based framework which allows one to solve composite optimization problems involving a Lipschitz-smooth function $f$ and a nonsmooth function $g$,

[TABLE]

The main idea is to replace $g$ with its Moreau envelope of index $\beta_{k}$ at each iteration $k$, with the index parameter $\beta_{k}$ going to [math]. This is equivalent to partial minimization with a quadratic penalization term, as in our algorithm. Like our algorithm, that of [37] is able to handle problems involving both smooth and nonsmooth terms, intersection of multiple constraint sets and affine constraints, however their algorithms employ different methods for these situations. Our algorithm uses an augmented Lagrangian to handle the affine constraint while the conditional gradient framework treats the affine constraint as a nonsmooth term $g$ and uses penalization to smooth the indicator function corresponding to the affine constraint. In particular circumstances, outlined in more detail in Section 6, our algorithms agree completely.

Another recent and parallel work to ours is that of [16], where the Frank-Wolfe via Augmented Lagrangian (FW-AL) is developed to approach the problem of minimizing a Lipschitz-smooth function over a convex, compact set with a linear constraint,

[TABLE]

The main idea of FW-AL is to use the augmented Lagrangian to handle the linear constraint and then apply the classical augmented Lagrangian algorithm, except that the marginal minimization on the primal variable that is usually performed is replaced by an inner loop of Frank-Wolfe. It turns out that the problem they consider is a particular case of ($\mathrsfs{P}$), discussed in Section 6.

1.4 Organization of the paper

In Section 2 we introduce the notation and review some necessary material from convex and real analysis. In Section 3 we present the Conditional Gradient with Augmented Lagrangian and Proximal-step (CGALP ) algorithm and the underlying assumptions. In Section 4, we first state our main convergence results and then turn to their proof. The latter is divided in three main parts. First we show the asymptotic feasibility, then the boundedness of the dual multiplier in the augmented Lagrangian and finally the optimality guarantees, i.e. weak convergence of the sequence $\left(\mu_{k}\right)_{k\in\mathbb{N}}$ to a solution of the dual problem, weak subsequential convergence of the sequence $\left(x_{k}\right)_{k\in\mathbb{N}}$ to a solution of the primal problem, and convergence of the Lagrangian values, and with convergence rates. In Section 5 we describe how our framework can be instantiated to solve a variety of composite optimization problems. In Section 6 we provide a more detailed discussion comparing CGALP to prior work. Some numerical results are reported in Section 7.

For readers who are primarily interested in the practical perspective, we suggest skipping directly to Section 3 for the algorithms and its assumptions or Section 4 for the main convergence results.

2 Notation and Preliminaries

We first recall some important definitions and results from convex analysis. For a more comprehensive coverage we refer the interested reader to [4, 30] and [33] in the finite dimensional case. Throughout, we let $\mathcal{H}$ denote an arbitrary real Hilbert space and $g$ an arbitrary function from $\mathcal{H}$ to the real extended line, namely $g:\mathcal{H}\to\mathbb{R}\cup\left\{+\infty\right\}$. The function $g$ is said to belong to $\Gamma_{0}\left(\mathcal{H}\right)$ if it is proper, convex, and lower semi-continuous. The domain of $g$ is defined to be $\mathrm{dom}\left(g\right)\stackrel{{\scriptstyle\text{\rm\tiny def}}}{{=}}\left\{x\in\mathcal{H}:g\left(x\right)<+\infty\right\}$. The Legendre-Fenchel conjugate of $g$ is the function $g^{*}:\mathcal{H}\to\mathbb{R}\cup\left\{+\infty\right\}$ such that, for every $u\in\mathcal{H}$,

[TABLE]

Notice that

[TABLE]

Moreau proximal mapping and envelope

The proximal operator for the function $g$ is defined to be

[TABLE]

and its Moreau envelope with parameter $\beta$ as

[TABLE]

Denoting $x^{+}=\mathrm{prox}_{g}\left(x\right)$, we have the following classical inequality (see, for instance, [30, Chapter 6.2.1]): for every $y\in\mathcal{H}$,

[TABLE]

We recall that the subdifferential of the function $g$ is defined as the set-valued operator $\partial g:\mathcal{H}\to 2^{\mathcal{H}}$ such that, for every $x$ in $\mathcal{H}$,

[TABLE]

We denote \mathrm{dom}(\partial g)\stackrel{{\scriptstyle\text{\rm\tiny def}}}{{=}}\big{\{}x\in\mathcal{H}:\;\partial g(x)\neq\emptyset\big{\}}. When $g$ belongs to $\Gamma_{0}\left(\mathcal{H}\right)$, it is well-known that the subdifferential is a maximal monotone operator. If, moreover, the function is Gâteaux differentiable at $x\in\mathcal{H}$, then $\partial g(x)=\left\{\nabla g(x)\right\}$. For $x\in\mathrm{dom}(\partial g)$, the minimal norm selection of $\partial g(x)$ is defined to be the unique element $\left\{\left[\partial g\left(x\right)\right]^{0}\right\}\stackrel{{\scriptstyle\text{\rm\tiny def}}}{{=}}\operatorname*{Argmin}\limits_{y\in\partial g\left(x\right)}\left\|y\right\|$. Then we have the following fundamental result about Moreau envelopes.

Proposition 2.1**.**

Given a function $g\in\Gamma_{0}\left(\mathcal{H}\right)$, we have the following:

1. (i)

The Moreau envelope is convex, real-valued, and continuous. 2. (ii)

Lax-Hopf formula: the Moreau envelope is the viscosity solution to the following Hamilton Jacobi equation:

[TABLE] 3. (iii)

The gradient of the Moreau envelope is $\frac{1}{\beta}$-Lipschitz continuous and is given by the expression

[TABLE] 4. (iv)

$\forall x\in\mathrm{dom}(\partial g)$, $\left\|\nabla g^{\beta}\left(x\right)\right\|\nearrow\left\|\left[\partial g\left(x\right)\right]^{0}\right\|$ as $\beta\searrow 0$. 5. (v)

$\forall x\in\mathcal{H}$, $g^{\beta}(x)\nearrow g(x)$ as $\beta\searrow 0$. In addition, given two positive real numbers $\beta^{\prime}<\beta$, for all $x\in\mathcal{H}$ we have

[TABLE]

• Proof.

(i): see [4, Proposition 12.15]. The proof for (ii) can be found in [2, Lemma 3.27 and Remark 3.32] (see also [19] or [1, Section 3.1]). The proof for claim (iii) can be found in [4, Proposition 12.29] and the proof for claim (iv) can be found in [4, Corollary 23.46]. For the first part in (v), see [4, Proposition 12.32(i)]. To show the first inequality in (v), combine (ii) and convexity of the function $\beta\mapsto g^{\beta}\left(x\right)$ for every $x\in\mathcal{H}$. The second inequality follows from the first one and (iv), taking the limit as $\beta^{\prime}\to 0$. ∎

Remark 2.2**.**

**

1. (i)

While the regularity claim in Proposition 2.1(iii)* of the Moreau envelope $g^{\beta}\left(x\right)$ w.r.t. $x$ is well-known, a less known result is the $C^{1}$-regularity w.r.t. $\beta$ for any $x\in\mathcal{H}$ (Proposition 2.1(ii)**). To our knowledge, the proof goes back, at least, to the book of [2]. Though it has been rediscovered in the recent literature in less general settings.* 2. (ii)

For given functions $H:\mathcal{H}\to\mathbb{R}$ and $g_{0}:\mathcal{H}\to\mathbb{R}$, a natural generalization of the Hamilton-Jacobi equation in (2.5) is

[TABLE]

Supposing that $H$ is convex and that $\lim\limits_{\left\|p\right\|\to+\infty}H(p)/\left\|p\right\|=+\infty$, the solution of the above system is given by the Lax-Hopf formula (see **[14, Theorem 5, Section 3.3.2]111The proof in [14] is given in the finite-dimensional case but it extends readily to any real Hilbert space.):

[TABLE]

If $H(p)=\frac{1}{2}\left\|p\right\|^{2}$, then $H^{*}(p)=\frac{1}{2}\left\|p\right\|^{2}$ and we recover the result in Proposition 2.1.

Regularity of differentiable functions

In what follows, we introduce some definitions related with regularity of differentiable functions. They will provide useful upper-bounds and descent properties. Notice that the the notions and results of this part are independent from convexity.

Definition 2.3**.**

($\omega$-smoothness) Consider a function $\omega:\mathbb{R}_{+}\to\mathbb{R}_{+}$ such that $\omega(0)=0$ and

[TABLE]

is non-decreasing. A differentiable function $g:\mathcal{H}\to\mathbb{R}$ is said to belong to $C^{1,\omega}\left(\mathcal{H}\right)$ or to be $\omega$-smooth if the following inequality is satisfied for every $x,y\in\mathcal{H}$:

[TABLE]

Lemma 2.4**.**

($\omega$-smooth Descent Lemma) Given a function $g\in C^{1,\omega}\left(\mathcal{H}\right)$ we have the following inequality: for every $x$ and $y$ in $\mathcal{H}$,

[TABLE]

where $\xi$ is defined in (2.6).

• Proof.

We recall here the simple proof for completeness:

[TABLE]

where in the first inequality we used Cauchy-Schwartz and in the second Definition 2.3. We conclude using the definition of $\xi$. ∎

For $L>0$ and $\omega\left(t\right)=Lt^{\nu}$, $\nu\in]0,1]$, $C^{1,\omega}\left(\mathcal{H}\right)$ is the space of differentiable functions with Hölder continuous gradients, in which case $\xi\left(s\right)=Ls^{\nu}/(1+\nu)$ and the Descent Lemma reads

[TABLE]

see e.g., [27, 28]. When $\nu=1$, we have that $C^{1,\omega}\left(\mathcal{H}\right)$ is the class of differentiable functions with $L$-Lipschitz continuous gradient, and one recovers the classical Descent Lemma.

Now, following [18], we introduce some notions that allow one to further generalize (2.7). Given a function $G:\mathcal{H}\to\mathbb{R}\cup\left\{+\infty\right\}$, differentiable on the open set $\mathcal{C}_{0}\subset\mathrm{int}\left(\mathrm{dom}\left(G\right)\right)$, define the Bregman divergence of $G$ as the function $D_{G}:\mathrm{dom}\left(G\right)\times\mathcal{C}_{0}\to\mathbb{R}$,

[TABLE]

Then we have the following result.

Lemma 2.5**.**

(Generalized Descent Lemma, [18, Lemma 1]) Let $G$ and $g$ be differentiable on $\mathcal{C}_{0}$, where $\mathcal{C}_{0}$ is an open subset of $\mathrm{int}\left(\mathrm{dom}\left(G\right)\right)$. Assume that $G-g$ is convex on $\mathcal{C}_{0}$. Then, for every $x$ and $y$ in $\mathcal{C}_{0}$,

[TABLE]

• Proof.

For our purpose, we intentionally weakened the hypothesis needed in the original result of [18, Lemma 1]. We repeat their argument but show the result is still valid under our weaker assumption. Let $x$ and $y$ be in $\mathcal{C}_{0}$, where, by hypothesis, $\mathcal{C}_{0}$ is open and contained in $\mathrm{int}\left(\mathrm{dom}\left(G\right)\right)$. As $G-g$ is convex and differentiable on $\mathcal{C}_{0}$, from the gradient inequality (2.4) we have, for all $y\in\mathcal{C}_{0}$,

[TABLE]

Rearranging the terms and using the definition of $D_{G}$ in (2.8), we obtain the claim. ∎

The previous lemma suggests the introduction of the following definition, which extends Definition 2.3.

Definition 2.6**.**

($\left(G,\zeta\right)$-smoothness) Let $G:\mathcal{H}\to\mathbb{R}\cup\left\{+\infty\right\}$ and $\zeta:]0,1]\to\mathbb{R}_{+}$. The pair $\left(g,\mathcal{C}\right)$, where $g:\mathcal{H}\to\mathbb{R}\cup\left\{+\infty\right\}$ and $\mathcal{C}\subset\mathrm{dom}(g)$, is said to be $\left(G,\zeta\right)$-smooth if there exists an open set $\mathcal{C}_{0}$ such that $\mathcal{C}\subset\mathcal{C}_{0}\subset\mathrm{int}\left(\mathrm{dom}\left(G\right)\right)$ and

1. (i)

$G$* and $g$ are differentiable on $\mathcal{C}_{0}$;* 2. (ii)

$G-g$* is convex on $\mathcal{C}_{0}$;* 3. (iii)

it holds

[TABLE]

$K_{\left(G,\zeta,\mathcal{C}\right)}$ is a far-reaching generalization of the standard curvature constant widely used in the literature of conditional gradient.

Remark 2.7**.**

Assume that $\left(g,\mathcal{C}\right)$ is $\left(G,\zeta\right)$-smooth. Using first Lemma 2.5 and then the definition in (2.9), we have the following descent property: for every $x,s\in\mathcal{C}$ and for every $\gamma\in]0,1]$,

[TABLE]

Notice that, as in the previous definition, we do not require $\mathcal{C}$ to be convex. So, in general, the point $z=x+\gamma\left(s-x\right)$ may not lie in $\mathcal{C}$.

Lemma 2.8**.**

Suppose that the set $\mathcal{C}$ is bounded and denote by $d_{\mathcal{C}}\stackrel{{\scriptstyle\text{\rm\tiny def}}}{{=}}\sup_{x,y\in\mathcal{C}}\left\|x-y\right\|$ its diameter. Moreover, assume that the function $g$ is $\omega$-smooth on some open and convex subset $\mathcal{C}_{0}$ containing $\mathcal{C}$. Set $\zeta(\gamma)\stackrel{{\scriptstyle\text{\rm\tiny def}}}{{=}}\xi(d_{\mathcal{C}}\gamma)$, where $\xi$ is given in (2.6). Then the pair $\left(g,\mathcal{C}\right)$ is $\left(g,\zeta\right)$-smooth with $K_{\left(g,\zeta,\mathcal{C}\right)}\leq d_{\mathcal{C}}$.

• Proof.

With $G=g$ and $g$ being $\omega$-smooth on $\mathcal{C}_{0}$, both $G$ and $g$ are differentiable on $\mathcal{C}_{0}$ and $G-g\equiv 0$ is convex on $\mathcal{C}_{0}$. Thus, all conditions required in Definition 2.6 hold true. It then remains to show (2.9) with the bound $K_{\left(g,\zeta,\mathcal{C}\right)}\leq d_{\mathcal{C}}$. First notice that, for every $x,s\in\mathcal{C}$ and for every $\gamma\in]0,1]$, the point $z=x+\gamma\left(s-x\right)$ belongs to $\mathcal{C}_{0}$. Indeed, $\mathcal{C}\subset\mathcal{C}_{0}$ and $\mathcal{C}_{0}$ is convex by hypothesis. In particular, as $g$ is $\omega$-smooth on $\mathcal{C}_{0}$, the Descent Lemma 2.4 holds between the points $x$ and $z$. Then

[TABLE]

In the first inequality we used Lemma 2.4, while in the second we used that $\left\|s-x\right\|\leq d_{\mathcal{C}}$ (both $x$ and $s$ belong to $\mathcal{C}$, that is bounded by hypothesis) and the monotonicity of the function $\xi$ (see Definition 2.3). ∎

Indicator and support functions

Given a subset $\mathcal{C}\subset\mathcal{H}$, we define its indicator function as $\iota_{\mathcal{C}}(x)=0$ if $x$ belongs to $\mathcal{C}$ and $\iota_{\mathcal{C}}(x)=+\infty$ otherwise. Recall that, if $\mathcal{C}$ is nonempty, closed, and convex, then $\iota_{\mathcal{C}}$ belongs to $\Gamma_{0}\left(\mathcal{H}\right)$. Remember also the definition of the support function of $\mathcal{C}$, $\sigma_{\mathcal{C}}\stackrel{{\scriptstyle\text{\rm\tiny def}}}{{=}}\iota_{\mathcal{C}}^{*}$. Equivalently, $\sigma_{\mathcal{C}}\left(x\right)\stackrel{{\scriptstyle\text{\rm\tiny def}}}{{=}}\sup\left\{\left\langle z,x\right\rangle:\ z\in\mathcal{C}\right\}$. We denote by $\mathrm{ri}\left(\mathcal{C}\right)$ the relative interior of the set $\mathcal{C}$ (in finite dimension, it is the interior for the topology relative to its affine full). We denote $\mathrm{par}(C)$ as the subspace parallel to $\mathcal{C}$ which, in finite dimension, takes the form $\mathbb{R}(C-C)$.

We have the following characterization of the support function from the relative interior in finite dimension.

Proposition 2.9**.**

([35, Lemma 1]) Let $\mathcal{H}$ be finite-dimensional and $\mathcal{C}\subset\mathcal{H}$ a nonempty, closed bounded and convex subset. If $0\in\mathrm{ri}(\mathcal{C})$, then $\sigma_{\mathcal{C}}\in\Gamma_{0}(\mathbb{R}^{n})$ is sublinear, non-negative and finite-valued, and

[TABLE]

Coercivity

We recall that a function $g$ is coercive if $\lim_{\left\|x\right\|\to+\infty}g\left(x\right)=+\infty$ and that coercivity is equivalent to the boundedness of the sublevel-sets [4, Proposition 11.11]. We have the following result, that relates coercivity to properties of the Fenchel conjugate.

Proposition 2.10**.**

([4, Theorem 14.17]) Given $g$ in $\Gamma_{0}\left(\mathcal{H}\right)$, $g^{*}$ is coercive if and only if $0\in\mathrm{int}\left(\mathrm{dom}(g)\right)$.

The recession function (sometimes referred to as the horizon function) of $g$ at a given point $d\in\mathbb{R}^{n}$ is defined to be $g^{d,\infty}:\mathbb{R}^{n}\to\mathbb{R}\cup\left\{+\infty\right\}$ such that, for every $x\in\mathbb{R}^{n}$,

[TABLE]

Recall that, if $g$ is convex, the recession function is independent from the selection of the point $d\in\mathbb{R}^{n}$ and can be then simply denoted as $g^{\infty}$. In finite dimension, the following result relates coercivity to properties of the recession function.

Proposition 2.11**.**

Let $g\in\Gamma_{0}\left(\mathbb{R}^{n}\right)$ and $A:\mathbb{R}^{m}\to\mathbb{R}^{n}$ be a linear operator. Then,

1. (i)

$g\mbox{ coercive }\iff g^{\infty}\left(x\right)>0\quad\forall x\neq 0$. 2. (ii)

$g^{\infty}\equiv\sigma_{\mathrm{dom}\left(g^{*}\right)}$. 3. (iii)

$\left(g\circ A\right)^{\infty}\equiv g^{\infty}\circ A$.

In particular, we deduce that $g\circ A$ is coercive if and only if $\sigma_{\mathrm{dom}\left(g^{*}\right)}(Ax)>0$ for every $x\neq 0$.

• Proof.

The proofs can be found in [34, Theorem 3.26], [34, Theorem 11.5] and [23, Corollary 3.2] respectively. ∎

Real sequences

We close this section with some definitions and lemmas for real sequences that will be used to prove the convergence properties of the algorithm. We denote $\ell_{+}$ as the set of all sequences in $[0,+\infty[$. Given $p\in[1,+\infty[$, $\ell^{p}$ is the space of real sequences $\left(r_{k}\right)_{k\in\mathbb{N}}$ such that $\left(\sum\limits_{k=1}^{\infty}|r_{k}|^{p}\right)^{1/p}<+\infty$. For $p=+\infty$, we denote by $\ell^{\infty}$ the space of bounded sequences. Furthermore, we will use the notation $\ell^{p}_{+}\stackrel{{\scriptstyle\text{\rm\tiny def}}}{{=}}\ell^{p}\cap\ell_{+}$. In the next, we recall some key results about real sequences.

Lemma 2.12**.**

([11, Lemma 3.1]) Consider three sequences $\left(r_{k}\right)_{k\in\mathbb{N}}\in\ell_{+}$, $\left(a_{k}\right)_{k\in\mathbb{N}}\in\ell_{+}$, and $\left(z_{k}\right)_{k\in\mathbb{N}}\in\ell_{+}^{1}$, such that

[TABLE]

Then $\left(r_{k}\right)_{k\in\mathbb{N}}$ is convergent and $\left(a_{k}\right)_{k\in\mathbb{N}}\in\ell^{1}_{+}$.

Lemma 2.13**.**

([36, Theorem 2] and [36, Proposition 2(ii)]) Consider two sequences $\left(p_{k}\right)_{k\in\mathbb{N}}\in\ell_{+}$ and $\left(w_{k}\right)_{k\in\mathbb{N}}\in\ell_{+}$ such that $\left(p_{k}w_{k}\right)_{k\in\mathbb{N}}\in\ell^{1}_{+}$ and $\left(p_{k}\right)_{k\in\mathbb{N}}\notin\ell^{1}$. Then the following holds:

1. (i)

there exists a subsequence $\left(w_{k_{j}}\right)_{j\in\mathbb{N}}$ such that

[TABLE]

where $P_{n}=\sum_{k=1}^{n}p_{k}$. In particular, $\liminf\limits_{k}w_{k}=0$. 2. (ii)

If moreover there exists a constant $\alpha>0$ such that $w_{k}-w_{k+1}\leq\alpha p_{k}$ for every $k\in\mathbb{N}$, then

[TABLE]

Lemma 2.14**.**

Consider the sequences $\left(r_{k}\right)_{k\in\mathbb{N}}\in\ell_{+}$, $\left(p_{k}\right)_{k\in\mathbb{N}}\in\ell_{+}$, $\left(w_{k}\right)_{k\in\mathbb{N}}\in\ell_{+}$, and $\left(z_{k}\right)_{k\in\mathbb{N}}\in\ell_{+}$. Suppose that $\left(z_{k}\right)_{k\in\mathbb{N}}\in\ell^{1}_{+}$, $\left(p_{k}\right)_{k\in\mathbb{N}}\notin\ell^{1}$, and that, for some $\alpha>0$, the following inequalities are satisfied for every $k\in\mathbb{N}$:

[TABLE]

Then,

1. (i)

$\left(r_{k}\right)_{k\in\mathbb{N}}$* is convergent and $\left(p_{k}w_{k}\right)_{k\in\mathbb{N}}\in\ell_{+}^{1}$.* 2. (ii)

$\lim\limits_{k}w_{k}=0$. 3. (iii)

For every $k\in\mathbb{N}$, $\inf_{1\leq i\leq k}w_{i}\leq(r_{0}+E)/P_{k}$, where, again, $P_{n}=\sum_{k=1}^{n}p_{k}$ and $E=\sum_{k=1}^{+\infty}z_{k}$. 4. (iv)

There exists a subsequence $\left(w_{k_{j}}\right)_{j\in\mathbb{N}}$ such that, for all $j\in\mathbb{N}$, $w_{k_{j}}\leq P_{k_{j}}^{-1}$.

• Proof.

1. (i)

See Lemma 2.12. 2. (ii)

Claim (ii) follows by combining (i) and Lemma 2.13(ii). 3. (iii)

Sum (2.10) using a telescoping property and summability of $\left(z_{k}\right)_{k\in\mathbb{N}}$. 4. (iv)

Claim (iv) follows by combining (i) and Lemma 2.13(i).

∎

Notice that the conclusions of Lemma 2.14 remain true if non-negativity of the sequence $\left(r_{k}\right)_{k\in\mathbb{N}}$ is replaced with the assumption that it is bounded from below by a trivial translation argument. Observe also that Lemma 2.14 guarantees the convergence of the whole sequence to zero, but it gives a convergence rate only on a subsequence.

3 Algorithm and assumptions

3.1 Algorithm

As described in the introduction, we combine penalization with the augmented Lagrangian approach to form the following functional

[TABLE]

where $\mu$ is the dual multiplier, and $\rho_{k}$ and $\beta_{k}$ are non-negative parameters. The steps of our scheme, then, are summarized in Algorithm 1.

For the interpretation of the algorithm, notice that the first step is equivalent to

[TABLE]

Now define the functional $\mathcal{E}_{k}\left(x,\mu\right)\stackrel{{\scriptstyle\text{\rm\tiny def}}}{{=}}f\left(x\right)+g^{\beta_{k}}\left(Tx\right)+\left\langle\mu,Ax-b\right\rangle+\frac{\rho_{k}}{2}\left\|Ax-b\right\|^{2}.$ By convexity of the set $\mathcal{C}$ and the definition of $x_{k+1}$ as a convex combination of $x_{k}$ and $s_{k}$, the sequence $\left(x_{k}\right)_{k\in\mathbb{N}}$ remains in $\mathcal{C}$ for all $k$, although the affine constraint $Ax_{k}=b$ might only be satisfied asymptotically. It is an augmented Lagrangian, where we do not consider the non-differentiable function $h$ and we replace $g$ by its Moreau envelope. Notice that

[TABLE]

where in the second equality we used 2.1(iii). Then $z_{k}$ is just $\nabla_{x}\mathcal{E}_{k}\left(x_{k},\mu_{k}\right)$ and the first three steps of the algorithm can be condensed in

[TABLE]

Thus the primal variable update of each step of our algorithm boils down to conditional gradient applied to the function $\mathcal{E}_{k}\left(\cdot,\mu_{k}\right)$, where the next iterate is a convex combination between the previous one and the new direction $s_{k}$. A standard update of the Lagrange multiplier $\mu_{k}$ follows.

3.2 Assumptions

3.2.1 Assumptions on the functions

In order to help the reading, we recall in a compact form the following notation that we will use to refer to various functionals throughout the paper:

[TABLE]

where $\overline{\rho}$ is defined in Assumption (P.4) to be $\overline{\rho}=\sup\limits_{k}\rho_{k}$.

In the list (3.4), we can recognize $\Phi$ as the objective, $\Phi_{k}$ as the smoothed objective augmented with a quadratic penalization of the constraint, and $\mathcal{L}_{k}$ as a smoothed augmented Lagrangian. $\mathcal{L}$ denotes the classical Lagrangian. Recall that $\left({x}^{\star},{\mu}^{\star}\right)\in\mathcal{H}_{p}\times\mathcal{H}_{d}$ is a saddle-point for the Lagrangian $\mathcal{L}$ if for every $\left(x,\mu\right)\in\mathcal{H}_{p}\times\mathcal{H}_{d}$,

[TABLE]

It is well-known from standard Lagrange duality, see e.g. [4, Proposition 19.19] or [30, Theorem 3.68], that the existence of a saddle point $\left({x}^{\star},{\mu}^{\star}\right)$ ensures strong duality, that ${x}^{\star}$ solves ($\mathrsfs{P}$) and ${\mu}^{\star}$ solves the dual problem,

[TABLE]

The following assumptions on the problem will be used throughout the convergence analysis (for some results only a subset of these assumptions will be needed):

1. (A.1)

$f,g\circ T$, and $h$ belong to $\Gamma_{0}\left(\mathcal{H}_{p}\right)$. 2. (A.2)

The pair $\left(f,\mathcal{C}\right)$ is $\left(F,\zeta\right)$-smooth (see Definition 2.6), where we recall $\mathcal{C}\stackrel{{\scriptstyle\text{\rm\tiny def}}}{{=}}\mathrm{dom}\left(h\right)$. 3. (A.3)

$\mathcal{C}$ is weakly compact (and thus contained in a ball of radius $R>0$). 4. (A.4)

$T\mathcal{C}\subset\mathrm{dom}(\partial g)$ and $\sup\limits_{x\in\mathcal{C}}\left\|\left[\partial g\left(Tx\right)\right]^{0}\right\|<\infty$. 5. (A.5)

$h$ is Lipschitz continuous relative to its domain $\mathcal{C}$ with constant $L_{h}\geq 0$, i.e., $\allowbreak\forall(x,z)\in\mathcal{C}^{2}$, $|h(x)-h(z)|\leq L_{h}\left\|x-z\right\|$. 6. (A.6)

There exists a saddle-point $\left({x}^{\star},{\mu}^{\star}\right)\in\mathcal{H}_{p}\times\mathcal{H}_{d}$ for the Lagrangian $\mathcal{L}$. 7. (A.7)

$\mathrm{ran}(A)$ is closed. 8. (A.8)

One of the following holds:

1. (a)

$A^{-1}\left(b\right)\cap\mathrm{int}\left(\mathrm{dom}\left(g\circ T\right)\right)\cap\mathrm{int}\left(\mathcal{C}\right)\neq\emptyset$, where $A^{-1}\left(b\right)$ is the pre-image of $b$ under $A$. 2. (b)

$\mathcal{H}_{p}$ and $\mathcal{H}_{d}$ are finite dimensional and

[TABLE]

At this stage, a few remarks are in order.

Remark 3.1**.**

**

1. (i)

By Assumption (A.1), $\mathcal{C}$ is also closed and convex. This together with Assumption (A.3) entail, upon using **[4, Lemma 3.29 and Theorem 3.32]**, that $\mathcal{C}$ is weakly compact. 2. (ii)

Since the sequence of iterates $\left(x_{k}\right)_{k\in\mathbb{N}}$ generated by Algorithm 1 is guaranteed to belong to $\mathcal{C}$ under (P.1), we have from (A.4)

[TABLE]

where $M$ is a positive constant. 3. (iii)

Assumption (A.5) will only be needed in the proof of convergence to optimality (Theorem 4.2). It is not needed to show asymptotic feasibility (Theorem 4.1). 4. (iv)

Assume that $A^{-1}(b)\cap\mathrm{dom}(g\circ T)\cap\mathcal{C}\neq\emptyset$, which entails that the set of minimizers of ($\mathrsfs{P}$) is a non-empty convex closed bounded set under (A.1)-(A.3). Then there are various domain qualification conditions, e.g., one of the conditions in **[4, Proposition 15.24 and Fact 15.25]**, that ensure the existence of a saddle-point for the Lagrangian $\mathcal{L}$ (see **[4, Theorem 19.1 and Proposition 9.19(v)]**). 5. (v)

Observe that under the inclusion assumption of Lemma 3.2, (A.8)(A.8)(a)* is equivalent to $A^{-1}\left(b\right)\cap\mathrm{int}\left(\mathcal{C}\right)\neq\emptyset$.* 6. (vi)

Assumption (A.8) will be crucial to show that $\bar{\varphi}$ is coercive on $\mathrm{ker}(A^{*})^{\bot}=\mathrm{ran}(A)$ (the last equality follows from (A.7)), and hence boundedness of the dual multiplier sequence $\left(\mu_{k}\right)_{k\in\mathbb{N}}$ provided by Algorithm 1 (see Lemma 4.10 and Lemma 4.11).

The uniform boundedness of the minimal norm selection on $\mathcal{C}$, as required in Assumption (A.4), is important when we will invoke Proposition 2.1(v) in our proofs to get meaningful estimates. The following result gives some sufficient conditions under which (A.4) holds (in fact an even stronger claim).

Lemma 3.2**.**

*Let $\mathcal{C}$ be a nonempty bounded subset of $\mathcal{H}_{p}$, $g\in\Gamma_{0}\left(\mathcal{H}_{v}\right)$ and $T:\mathcal{H}_{p}\to\mathcal{H}_{v}$ be a bounded linear operator. Suppose that $T\mathcal{C}\subset\mathrm{int}\left(\mathrm{dom}\left(g\right)\right)$. Then the assumption (A.4) holds. *

• Proof.

Since $g\in\Gamma_{0}\left(\mathcal{H}_{p}\right)$, it follows from [4, Proposition 16.21] that

[TABLE]

Moreover, by [4, Corollary 8.30(ii) and Proposition 16.14], we have that $\partial g$ is locally weakly compact on $\mathrm{int}\left(\mathrm{dom}\left(g\right)\right)$. In particular, as we assume that $\mathcal{C}$ is bounded, so is $T\mathcal{C}$, and since $T\mathcal{C}\subset\mathrm{int}\left(\mathrm{dom}\left(g\right)\right)$, it means that for each $z\in T\mathcal{C}$ there exists an open neighborhood of $z$, denoted by $U_{z}$, such that $\partial g\left(U_{z}\right)$ is bounded. Since $\left(U_{z}\right)_{z\in\mathcal{C}}$ is an open cover of $T\mathcal{C}$ and $T\mathcal{C}$ is bounded, there exists a finite subcover $\left(U_{z_{k}}\right)_{k=1}^{n}$. Then,

[TABLE]

Since the right-hand-side is bounded (as it is a finite union of bounded sets),

[TABLE]

whence the desired conclusion trivially follows. ∎

3.2.2 Assumptions on the parameters

We also use the following assumptions on the parameters of Algorithm 1 (recall the function $\zeta$ in Definition 2.6):

1. (P.1)

$\left(\gamma_{k}\right)_{k\in\mathbb{N}}\subset]0,1]$ and the sequences $\left(\zeta\left(\gamma_{k}\right)\right)_{k\in\mathbb{N}},\left(\gamma_{k}^{2}/\beta_{k}\right)_{k\in\mathbb{N}}$ and $\left(\gamma_{k}\beta_{k}\right)_{k\in\mathbb{N}}$ belong to $\ell^{1}_{+}$. 2. (P.2)

$\left(\gamma_{k}\right)_{k\in\mathbb{N}}\notin\ell^{1}$. 3. (P.3)

$\left(\beta_{k}\right)_{k\in\mathbb{N}}\in\ell_{+}$ is non-increasing and converges to [math]. 4. (P.4)

$\left(\rho_{k}\right)_{k\in\mathbb{N}}\in\ell_{+}$ is non-decreasing with $0<\underline{\rho}=\inf_{k}\rho_{k}\leq\sup_{k}\rho_{k}=\overline{\rho}<+\infty$. 5. (P.5)

For some positive constants $\underline{M}$ and $\overline{M}$, $\underline{M}\leq\inf_{k}\left(\gamma_{k}/\gamma_{k+1}\right)\leq\sup_{k}\left(\gamma_{k}/\gamma_{k+1}\right)\leq\overline{M}$. 6. (P.6)

$\left(\theta_{k}\right)_{k\in\mathbb{N}}$ satisfies $\theta_{k}=\frac{\gamma_{k}}{c}$ for all $k\in\mathbb{N}$ for some $c>0$ such that $\frac{\overline{M}}{c}-\frac{\underline{\rho}}{2}<0$. 7. (P.7)

$\left(\gamma_{k}\right)_{k\in\mathbb{N}}$ and $\left(\rho_{k}\right)_{k\in\mathbb{N}}$ satisfy $\rho_{k+1}-\rho_{k}-\gamma_{k+1}\rho_{k+1}+\frac{2}{c}\gamma_{k}-\frac{\gamma_{k}^{2}}{c}\leq\gamma_{k+1}$ for all $k\in\mathbb{N}$ and for $c$ in (P.6).

Remark 3.3**.**

**

1. (i)

One can recognize that the update of the dual multiplier $\mu_{k}$ in Algorithm 1 has a flavour of gradient ascent applied to the augmented dual with step-size $\theta_{k}$. However, unlike the standard method of multipliers with the augmented Lagrangian, Assumption (P.6) requires $\theta_{k}$ to vanish in our setting. The underlying reason is that our update can be seen as an inexact dual ascent (i.e., exactness stems from the conditional gradient-based update on $x_{k}$ which is not a minimization of over $x$ of the augmented Lagrangian $\mathcal{L}_{k}$). Thus $\theta_{k}$ must annihilate this error asymptotically. 2. (ii)

A sufficient condition for (P.7) to hold consists of taking $\rho_{k}\equiv\rho>0$ and $\gamma_{k+1}\geq\frac{2}{c(1+\rho)}\gamma_{k}$. In particular, if $\left(\gamma_{k}\right)_{k\in\mathbb{N}}$ satisfies (P.5), then, for (P.7) to hold, it is sufficient to take $\rho_{k}\equiv\rho>2\overline{M}/c$ as supposed in (P.6). 3. (iii)

The relevance of having $\rho_{k}$ vary is that it allows for more general and less stringent choice of the step-size $\gamma_{k}$. It is, however, possible (and easier in practice), to simply pick $\rho_{k}\equiv\rho$ for all $k\in\mathbb{N}$ as described above.

There is a large class of sequences that fulfill the requirements (P.1)-(P.7). A typical one is as follows.

Example 3.4**.**

Take222Of course, one can add a scaling factor in the choice of the parameters which would allow for more practical flexibility. But this does not change anything to our discussion nor to the bahviour of the CGALP algorithm for $k$ large enough., for $k\in\mathbb{N}$,

[TABLE]

In this case, one can take the crude bounds $\underline{M}=(\log(2)/\log(3))^{a}$ and $\overline{M}=2^{1-b}$, and choose $\rho>2\overline{M}/c$ as devised in Remark 3.3(ii)**. In turn, (P.4)-(P.7) hold. In addition, suppose that $f$ has a $\nu$-Hölder continuous gradient (see (2.7)). Thus for (P.1)-(P.2) to hold, simple algebra shows that the allowable choice of $b$ is in $\left[0,\min\left({1/3,\frac{\nu}{1+\nu}}\right)\right[$.

4 Convergence analysis

4.1 Main results

We state here our main results.

Theorem 4.1** (Asymptotic feasibility).**

Suppose that Assumptions (A.1)-(A.4) and (A.6) hold. Consider the sequence of iterates $\left(x_{k}\right)_{k\in\mathbb{N}}$ from Algorithm 1 with parameters satisfying Assumptions (P.1)-(P.6). Then,

1. (i)

$Ax_{k}$* converges strongly to $b$ as $k\to\infty$, i.e., the sequence $\left(x_{k}\right)_{k\in\mathbb{N}}$ is asymptotically feasible for ($\mathrsfs{P}$) in the strong topology.* 2. (ii)

Pointwise rate:

[TABLE]

where, for all $k\in\mathbb{N}$, $\Gamma_{k}\stackrel{{\scriptstyle\text{\rm\tiny def}}}{{=}}\sum_{i=0}^{k}\gamma_{i}$. 3. (iii)

Ergodic rate: for each $k\in\mathbb{N}$, let $\bar{x}_{k}\stackrel{{\scriptstyle\text{\rm\tiny def}}}{{=}}\sum_{i=0}^{k}\gamma_{i}x_{i}/\Gamma_{k}$. Then

[TABLE]

Theorem 4.1 will be proved in Section 4.3.

Theorem 4.2** (Convergence to optimality).**

Suppose that assumptions (A.1)-(A.8) and (P.1)-(P.7) hold, with $\underline{M}\geq 1$. Let $\left(x_{k}\right)_{k\in\mathbb{N}}$ be the sequence of primal iterates generated by Algorithm 1 and $({x}^{\star},{\mu}^{\star})$ a saddle-point pair for the Lagrangian. Then, in addition to the results of Theorem 4.1, the following holds

1. (i)

Convergence of the Lagrangian:

[TABLE] 2. (ii)

Every weak cluster point $\bar{x}$ of $\left(x_{k}\right)_{k\in\mathbb{N}}$ is a solution of the primal problem ($\mathrsfs{P}$), and $\left(\mu_{k}\right)_{k\in\mathbb{N}}$ converges weakly to $\bar{\mu}$ a solution of the dual problem ($\mathrsfs{D}$), i.e., $(\bar{x},\bar{\mu})$ is a saddle point of $\mathcal{L}$. 3. (iii)

Pointwise rate:

[TABLE] 4. (iv)

Ergodic rate: for each $k\in\mathbb{N}$, let $\bar{x}_{k}\stackrel{{\scriptstyle\text{\rm\tiny def}}}{{=}}\sum_{i=0}^{k}\gamma_{i}x_{i+1}/\Gamma_{k}$. Then

[TABLE]

An important observation is that Theorem 4.2, which will be proved in Section 4.5, actually shows that

[TABLE]

and subsequentially, for each $j\in\mathbb{N}$,

[TABLE]

This means, in particular, that the pointwise rate for feasibility and optimality hold simulatenously for the same subsequence.

The following corollary is immediate.

Corollary 4.3**.**

*Under the assumptions of Theorem 4.2, if the problem ($\mathrsfs{P}$) admits a unique solution ${x}^{\star}$, then the primal-dual pair sequence $\left(x_{k},\mu_{k}\right)_{k\in\mathbb{N}}$ converges weakly to a saddle point $({x}^{\star},{\mu}^{\star})$. *

• Proof.

By uniqueness, it follows from Theorem 4.2(ii) that $\left(x_{k}\right)_{k\in\mathbb{N}}$ has exactly one weak sequential cluster point which is the solution to ($\mathrsfs{P}$). Weak convergence of the sequence $\left(x_{k}\right)_{k\in\mathbb{N}}$ then follows from [4, Lemma 2.38]. ∎

Example 4.4**.**

Suppose that the sequences of parameters are chosen according to Example 3.4. Let the function $\sigma:t\in\mathbb{R}^{+}\mapsto(\log(t+2))^{a}/(t+1)^{1-b}$. We obviously have $\sigma(k)=\gamma_{k}$ for $k\in\mathbb{N}$. Moreover, it is easy to see that $\exists k^{\prime}\geq 0$ (depending on $a$ and $b$), such that $\sigma$ is decreasing for $t\geq k^{\prime}$. Thus, $\forall k\geq k^{\prime}$, we have

[TABLE]

It is easy to show, using integration by parts for the first case, that

[TABLE]

This result reveals that picking $a$ and $b$ as large as possible results in a faster convergence rate, with the proviso that $b$ satisfy some conditions for (P.1)-(P.7) to hold, see the discussion in Example 3.4 for the largest possible choice of $b$.

4.2 Preparatory results

The next result is a direct application of the Descent Lemma 2.7 and the generalized one in Lemma 2.5 to the specific case of Algorithm 1. It allows to obtain a descent property for the function $\mathcal{E}_{k}\left(\cdot,\mu_{k}\right)$ between the previous iterate $x_{k}$ and next one $x_{k+1}$.

Lemma 4.5**.**

Suppose Assumptions (A.1), (A.2) and (P.1) hold. For each $k\in\mathbb{N}$, define the quantity

[TABLE]

Then, for each $k\in\mathbb{N}$, we have the following inequality:

[TABLE]

• Proof.

Define for each $k\in\mathbb{N}$,

[TABLE]

so that $\mathcal{E}_{k}\left(x,\mu\right)=f(x)+\tilde{\mathcal{E}}_{k}\left(x,\mu\right)$. Compute

[TABLE]

which is Lipschitz-continuous with constant $L_{k}=\frac{\|T\|^{2}}{\beta_{k}}+\|A\|^{2}\rho_{k}$ by virtue of (A.1) and Proposition 2.1(iii). Then we can use the Descent Lemma (2.7) with $\nu=1$ on $\tilde{\mathcal{E}}_{k}\left(\cdot,\mu_{k}\right)$ between the points $x_{k}$ and $x_{k+1}$, to obtain, for each $k\in\mathbb{N}$,

[TABLE]

From Assumption (A.2), Lemma 2.5 and Remark 2.7, we have, for each $k\in\mathbb{N}$,

[TABLE]

where we used that both $x_{k}$ and $s_{k}$ lie in $\mathcal{C}$, that $\gamma_{k}$ belongs to $]0,1]$ by (P.1) and thus $x_{k+1}=x_{k}+\gamma_{k}\left(s_{k}-x_{k}\right)\in\mathcal{C}$. Summing (4.8) with the latter and recalling that $\mathcal{E}_{k}\left(x,\mu_{k}\right)=f(x)+\tilde{\mathcal{E}}_{k}\left(x,\mu_{k}\right)$, we obtain the claim. ∎

Again for the function $\mathcal{E}_{k}\left(\cdot,\mu_{k}\right)$, we also have a lower-bound, presented in the next lemma.

Lemma 4.6**.**

Suppose Assumptions (A.1) and (A.2) hold. Then, for all $k\in\mathbb{N}$, for all $x,x^{\prime}\in\mathcal{H}_{p}$ and for all $\mu\in\mathcal{H}_{d}$,

[TABLE]

• Proof.

First, split the function $\mathcal{E}_{k}\left(\cdot,\mu\right)$ as $\mathcal{E}_{k}\left(x,\mu\right)=\mathcal{E}_{k}^{0}\left(x,\mu\right)+\frac{\rho_{k}}{2}\|Ax-b\|^{2}$ for an opportune definition of $\mathcal{E}_{k}^{0}\left(\cdot,\mu\right)$. For the first term, simply by convexity, we have

[TABLE]

Now use the strong convexity of the term $\left(\rho_{k}/2\right)\|\cdot-b\|^{2}$ between points $Ax$ and $Ax^{\prime}$, to affirm that

[TABLE]

Compute

[TABLE]

Summing (4.9) and (4.10) and invoking the gradient computation above, we obtain the claim. ∎

Lemma 4.7**.**

Suppose that assumptions (A.1)-(A.8) and (P.1)-(P.7) hold, with $\underline{M}\geq 1$. Let $\left(x_{k}\right)_{k\in\mathbb{N}}$ be the sequence of primal iterates generated by Algorithm 1 and ${\mu}^{\star}$ a solution of the dual problem ($\mathrsfs{D}$). Then we have the following estimate,

[TABLE]

• Proof.

First define $u_{k}\stackrel{{\scriptstyle\text{\rm\tiny def}}}{{=}}\left[\partial g(Tx_{k})\right]^{0}$ and recall that, by (A.4) and its consequence in (3.7), $\|u_{k}\|\leq M$ for every $k\in\mathbb{N}$. Then,

[TABLE]

where we used the subdifferential inequality (2.4) on $g$, the gradient inequality on $f$, the $L_{h}$-Lipschitz continuity of $h$ relative to $\mathcal{C}$ (see (A.5)), and the Cauchy-Schwartz inequality on the scalar product. Since $x_{k+1}=x_{k}+\gamma_{k}\left(x_{k}-s_{k}\right)$, we obtain

[TABLE]

where we have denoted by $D$ the constant $D\stackrel{{\scriptstyle\text{\rm\tiny def}}}{{=}}\sup_{x\in\mathcal{C}}\|\nabla f(x)\|<+\infty$ (see  (4.32)). ∎

Lemma 4.8**.**

Suppose that assumptions (A.3) and (P.4) hold. Let $\left(x_{k}\right)_{k\in\mathbb{N}}$ be the sequence of primal iterates generated by Algorithm 1. Then we have the following estimate,

[TABLE]

where $R$ is the radius of the ball containing $\mathcal{C}$ and $\overline{\rho}=\sup_{k}\rho_{k}$.

• Proof.

By (P.4) and convexity of the function $\frac{\rho_{k+1}}{2}\|A\cdot-b\|^{2}$, we have

[TABLE]

Now compute the gradient and use the definition of $x_{k+1}$, to obtain

[TABLE]

In the last inequality, we used Cauchy-Schwartz inequality, triangle inequality, the fact that

$\left\|x_{k}-s_{k}\right\|\leq d_{\mathcal{C}}$, and assumptions (A.3) and (P.4) (respectively, $\sup_{x\in\mathcal{C}}\|x\|\leq R$ and $\rho_{k+1}\leq\overline{\rho}$). ∎

4.3 Asymptotic feasibility

We begin with an intermediary lemma establishing the main feasibility estimation and some summability results that will also be used in the proof of optimality.

Lemma 4.9**.**

Suppose that Assumptions (A.1)-(A.4) and (A.6) hold. Consider the sequence of iterates $\left(x_{k}\right)_{k\in\mathbb{N}}$ from Algorithm 1 with parameters satisfying Assumptions (P.1)-(P.6). Define the two quantities $\Delta^{p}_{k}$ and $\Delta^{d}_{k}$ in the following way,

[TABLE]

where we have denoted $\tilde{\mathcal{L}}_{k}\left(\mu_{k}\right)\stackrel{{\scriptstyle\text{\rm\tiny def}}}{{=}}\min_{x}\mathcal{L}_{k}\left(x,\mu_{k}\right)$ and $\tilde{\mathcal{L}}\stackrel{{\scriptstyle\text{\rm\tiny def}}}{{=}}\mathcal{L}\left({x}^{\star},{\mu}^{\star}\right)$. Denote the sum $\Delta_{k}\stackrel{{\scriptstyle\text{\rm\tiny def}}}{{=}}\Delta_{k}^{p}+\Delta_{k}^{d}$. Then we have the following estimation,

[TABLE]

and, moreover,

[TABLE]

• Proof.

First notice that the quantity $\Delta^{p}_{k}\geq 0$ and can be seen as a primal gap at iteration $k$ while $\Delta^{d}_{k}$ may be negative but is bounded from below by our assumptions. Indeed, in view of (A.1), (A.6) and Remark 3.1(iv), $\tilde{\mathcal{L}}_{k}\left(\mu_{k}\right)$ is bounded from above since

[TABLE]

where we used Proposition 2.1(v) in the last inequality.

We denote a minimizer of $\mathcal{L}_{k}\left(x,\mu_{k}\right)$ by $\tilde{x}_{k}\in\operatorname*{Argmin}\limits_{x\in\mathcal{H}_{p}}\mathcal{L}_{k}\left(x,\mu_{k}\right)$, which exists and belongs to $\mathcal{C}$ by (A.1)-(A.3). Then, we have

[TABLE]

Since $\tilde{x}_{k}$ is a minimizer of $\mathcal{L}_{k}\left(x,\mu_{k}\right)$ we have that $\mathcal{L}_{k}\left(\tilde{x}_{k},\mu_{k}\right)\leq\mathcal{L}_{k}\left(\tilde{x}_{k+1},\mu_{k}\right)$ which leads to,

[TABLE]

where the last inequality comes from Proposition 2.1(v) and the assumptions (P.3) and (P.4). Combining this with (4.11),

[TABLE]

where in the last equality we used the definition of $\mu_{k+1}$. Meanwhile, for the primal gap we have

[TABLE]

Note that

[TABLE]

and estimate $\mathcal{L}_{k}\left(\tilde{x}_{k},\mu_{k}\right)-\mathcal{L}_{k+1}\left(\tilde{x}_{k+1},\mu_{k+1}\right)$ as in (4.12), to get

[TABLE]

Using (4.12) and (4.13), we then have

[TABLE]

Note that

[TABLE]

Then

[TABLE]

We denote by $\textbf{T1}=\mathcal{L}_{k+1}\left(x_{k+2},\mu_{k+1}\right)-\mathcal{L}_{k+1}\left(x_{k+1},\mu_{k+1}\right)$ and the remaining part of the right-hand side by T2. For the moment, we focus our attention on T1. Recall that $\mathcal{L}_{k}\left(x,\mu_{k}\right)=\mathcal{E}_{k}\left(x,\mu_{k}\right)+h\left(x\right)$ and apply Lemma 4.5 between points $x_{k+2}$ and $x_{k+1}$, to get

[TABLE]

By (A.1) we have that $h$ is convex and thus, since $x_{k+2}$ is a convex combination of $x_{k+1}$ and $s_{k+1}$, we get

[TABLE]

Applying the definition of $s_{k}$ as the minimizer of the linear minimization oracle and Lemma 4.6 at the points $\tilde{x}_{k+1}$, $x_{k+1}$, and $\mu_{k+1}$ gives,

[TABLE]

where we used that $\tilde{x}_{k+1}$ is a minimizer of $\mathcal{L}_{k+1}\left(\cdot,\mu_{k+1}\right)$ in the last inequality. Now, combining T1 and T2 and using the Pythagoras identity we have

[TABLE]

Under (P.6) we have $\theta_{k}=\frac{\gamma_{k}}{c}$ for some $c>0$ such that

[TABLE]

where $\overline{M}$ is the constant such that $\gamma_{k}\leq\overline{M}\gamma_{k+1}$ (see Assumption (P.5)). Then, using (P.5) and the above inequality,

[TABLE]

Now use the fact that $x_{k+2}=x_{k+1}+\gamma_{k+1}\left(s_{k+1}-x_{k+1}\right)$ to estimate

[TABLE]

Moreover, by the two assumptions (P.3), (A.4) and Proposition 2.1(v), (3.7) holds with a constant $M>0$ , and thus with Proposition 2.1(iv) we obtain

[TABLE]

Plugging (4.15), (4.16) and (4.17) into (4.14), we get

[TABLE]

Because of the assumptions (P.1) and (P.4), and in view of the definition of $L_{k}$ in (4.7), we have the following,

[TABLE]

For the telescopic terms from the right hand side of (4.18) we have

[TABLE]

where $R$ is the constant arising from (A.3). Under (P.1) we also have that

[TABLE]

Using the notation of Lemma 2.14, we set

[TABLE]

We have shown above that

[TABLE]

where $\left(z_{k}\right)_{k\in\mathbb{N}}\in\ell_{+}^{1}$, and $r_{k}$ is bounded from below. We then deduce using Lemma 2.14(i) that $\left(r_{k}\right)_{k\in\mathbb{N}}$ is convergent and

[TABLE]

Consequently,

[TABLE]

since, by Jensen’s inequality,

[TABLE]

∎

We are now ready to prove Theorem 4.1, i.e., to show that the sequence of iterates $\left(x_{k}\right)_{k\in\mathbb{N}}$ is asymptotically feasible.

• Proof.

1. (i)

By Lemma 4.8 with $\rho_{k}\equiv\rho_{k+1}\equiv 2$, we have

[TABLE]

Using this together with Lemma 4.9 and Assumption (P.2), we are in position to apply Lemma 2.14(ii) to conclude that $\lim_{k\to\infty}\left\|Ax_{k}-b\right\|=0$. 2. (ii)

The rates in (4.1) follow respectively from Lemma 2.14(iii) and Lemma 2.14(iv). 3. (iii)

We have, by Jensen’s inequality and Lemma 4.9, that

[TABLE]

∎

4.4 Dual multiplier boundedness

In this part we provide a lemma that shows the sequence of dual variables $\left(\mu_{k}\right)_{k\in\mathbb{N}}$ generated by Algorithm 1 is bounded.

We start by studying coercivity of $\bar{\varphi}$.

Lemma 4.10**.**

Suppose that Assumptions (A.1)-(A.3) and (A.6)-(A.8) hold. Then $\bar{\varphi}$ is coercive on $\mathrm{ran}\left(A\right)$.

• Proof.

From (3.4), we have, for any $c\in A^{-1}(b)$, that

[TABLE]

Moreover, Assumptions (A.1) and (A.7) entail that $\bar{\Phi}\in\Gamma_{0}(\mathcal{H}_{p})$. We now consider separately the two assumptions.

1. (a)

Case of (A.8)(A.8)(a): If follows from the Fenchel-Moreau theorem ([4, Theorem 13.32]) that

[TABLE]

Using this, together with Proposition 2.10 and (A.2), we can assert that $\bar{\Phi}^{*}-\left\langle c,\cdot\right\rangle$ is coercive if and only if

[TABLE]

But this is precisely what (A.8)(A.8)(a) guarantees. In turn, using [4, Proposition 14.15], (A.8)(A.8)(a) is equivalent to

[TABLE]

Using standard results on linear operators in Hilbert spaces [4, Facts 2.18 and 2.19], we have

[TABLE]

Combining the last two inequalities, we deduce that under (A.8)(A.8)(a),

[TABLE]

which in turn is equivalent to coercivity of $\bar{\varphi}$ on $\mathrm{ran}(A)$ by [4, Proposition 14.15]. 2. (b)

Case of (A.8)(A.8)(b): Since $\mathcal{H}_{d}$ is finite dimensional, We have, $\forall u\in\mathcal{H}_{d}$,

[TABLE]

Notice that, by Assumption (A.4), we have $\mathrm{dom}\left(g\circ T\right)\cap\mathcal{C}=\mathcal{C}$. Thus, using Proposition 2.11(i), we have the following chain of equivalences

[TABLE]

For this to hold, and since $\mathrm{ran}\left(A\right)=\mathrm{ker}\left(A^{*}\right)^{\bot}$, a sufficient condition is that

[TABLE]

It remains to check that the latter condition holds under (A.8)(A.8)(b). First, observe that $\mathcal{C}$ is a nonempty bounded convex set thanks to (A.1) and (A.3). The first condition in (A.8)(A.8)(b) is equivalent to $0\in\mathrm{ri}(\mathcal{C}-c)$ for some $c\in A^{-1}(b)$. It then follows from Proposition 2.9 that

[TABLE]

which then implies (4.21) thanks to the second condition in (A.8)(A.8)(b).

∎

Lemma 4.11**.**

Suppose that assumptions (A.1)-(A.3) and (A.6)-(A.8) and (P.1)-(P.6) hold. Then the sequence of dual iterates $\left(\mu_{k}\right)_{k\in\mathbb{N}}$ generated by Algorithm 1 is bounded.

• Proof.

Using the notation in (3.4), the primal problem:

[TABLE]

is obviously equivalent to

[TABLE]

We associate to the previous the following regularized primal problem:

[TABLE]

and its Lagrangian dual, namely:

[TABLE]

Now consider the dual function in the latter, namely $\varphi_{k}(\mu)\stackrel{{\scriptstyle\text{\rm\tiny def}}}{{=}}-\inf_{x\in\mathcal{H}_{p}}\ \mathcal{L}_{k}\left(x,\mu\right)$. Observe that the minimum is actually attained owing to (A.1) and (A.3). Now we claim that $\varphi_{k}$ is continuously differentiable with $L_{\nabla\varphi_{k}}$-Lipschitz gradient, and $1/\underline{\rho}$ (see (P.4)) is an upper-bound for $\left(L_{\nabla\varphi_{k}}\right)_{k\in\mathbb{N}}$. In order to show it, introduce the notation

[TABLE]

By definition, we have

[TABLE]

Using Fenchel-Rockafellar duality and strong duality, which holds by (P.4) and continuity of $G_{k}$ (see, for instance, [30, Theorem 3.51]), we have the following equality,

[TABLE]

where we have used the fact that the conjugate of a linear perturbation is the translation of the conjugate in the last line. Substituting the above into (4.22) we find

[TABLE]

Moreover, from the primal-dual extremality relationships [30, Theorem 3.51(i)], we have

[TABLE]

where $\tilde{x}$ is a minimizer (which exists and belongs to $\mathcal{C}$) of the primal objective $\mathcal{L}_{k}\left(\cdot,\mu\right)$ and $\tilde{v}$ is the unique minimizer to the associated dual objective. Now, using the change of variable $u=v+\mu$, we get

[TABLE]

where the notation $\left[\cdot\right]^{\rho_{k}}$ denotes the Moreau envelope with parameter $\rho_{k}$ as defined in (2.2). It follows from Proposition 2.1(i) and (iii), that $\varphi_{k}$ is convex, real-valued and its gradient, given by

[TABLE]

is $1/\rho_{k}$-Lipschitz continuous since the gradient of a Moreau envelope with parameter $\rho_{k}$ is $1/\rho_{k}$-Lipschitz continuous (see Proposition 2.1(iii)). As $\rho_{k}$ is non-decreasing, $1/\rho_{k}\leq 1/\underline{\rho}$ and the sequence of functions $\left(\nabla\varphi_{k}\right)_{k\in\mathbb{N}}$ is uniformly Lipschitz-continuous with constant $1/\underline{\rho}$. In addition, combining (4.23) and (4.24), and recalling the change of variable $\tilde{u}=\tilde{v}+\mu$, we get that

[TABLE]

As in Lemma 4.9, we are going to denote $\tilde{x}_{k}$ a minimizer of $\mathcal{L}_{k}\left(x,\mu_{k}\right)$. Then, from the Descent Lemma (see Proposition 2.4 and inequality (2.7)), we have

[TABLE]

Now substitute in the right-hand-side the expression $\nabla\varphi_{k}(\mu_{k})=A\tilde{x}_{k}-b$ in (4.25) and the update $\mu_{k+1}=\mu_{k}+\theta_{k}\left(Ax_{k+1}-b\right)$ from the algorithm, to obtain

[TABLE]

where we estimated the scalar product by Cauchy-Schwartz and Young inequality. Moreover, by definition,

[TABLE]

Now recall assumptions (P.3) and (P.4): for $\beta_{k}$ non-increasing, $\left[g^{\beta_{k}}-g^{\beta_{k+1}}\right](Tx)\leq 0$ for every $x\in\mathcal{H}_{p}$ by Proposition 2.1(v) and, for $\rho_{k}$ non-decreasing, $\rho_{k}-\rho_{k+1}\leq 0$. Then we can estimate the right-hand-side of (4.27) to obtain

[TABLE]

Sum (4.26) with the latter, to obtain

[TABLE]

By Assumption (P.6), $\theta_{k}=\gamma_{k}/c$ where $\gamma_{k}\leq 1$. Moreover, by assumption (P.5), $\gamma_{k}\leq\overline{M}\gamma_{k+1}$. Then,

[TABLE]

Notice that the right-hand-side is in $\ell^{1}_{+}$, because both $\left(\gamma_{k}\left\|Ax_{k}-b\right\|^{2}\right)_{k\in\mathbb{N}}$ and $\left(\gamma_{k}\left\|A\tilde{x}_{k}-b\right\|^{2}\right)_{k\in\mathbb{N}}$ are in $\ell^{1}_{+}$ by Lemma 4.9. Additionally, $\left(\varphi_{k}(\mu_{k})\right)_{k\in\mathbb{N}}$ is bounded from below. Indeed, by virtue of (A.6) and Remark 3.1(iv), we have

[TABLE]

Then we can use Lemma 2.14(i) on inequality (4.28) to conclude that $\left(\varphi_{k}\left(\mu_{k}\right)\right)_{k\in\mathbb{N}}$ is convergent and, in particular, bounded. Now recall $\Phi_{k}$, $\bar{\Phi}$ and $\bar{\varphi}$ from (3.4). Notice that

[TABLE]

It then follows that

[TABLE]

where we used Proposition 2.1(v) and the fact in (2.1). We are now in position to invoke Lemma 4.10 which shows that $\bar{\varphi}$ is coercive on $\mathrm{ran}(A)$, and thus, by (4.29), $\left(\varphi_{k}\right)_{k\in\mathbb{N}}$ is coercive uniformly in $k$ on $\mathrm{ran}(A)$. In turn, since $\mathrm{ran}(A)$ is closed and $\left(\mu_{k}\right)_{k\in\mathbb{N}}\subset\mathrm{ran}(A)=\mathrm{ker}(A^{*})^{\bot}$, we have from (4.29) and the proof of Lemma 4.10 that

[TABLE]

which shows that $\left(\mu_{k}\right)_{k\in\mathbb{N}}$ is indeed bounded by boundedness of $\left(\varphi_{k}\left(\mu_{k}\right)\right)_{k\in\mathbb{N}}$. ∎

4.5 Optimality

In this section we prove Theorem 4.2 by establishing convergence of the Lagrangian values to the optimum (i.e., the value at the saddle-point).

We start by showing some boundedness claims that will be important in our proof.

Lemma 4.12**.**

Under assumptions (A.1)-(A.8) and (P.1)-(P.6), the objective $\Phi$ is bounded on $\mathcal{C}$, and thus

[TABLE]

where we recall the radius $R$ from assumption (A.3).

• Proof.

By assumption (A.4), $g$ is subdifferentiable at $Tx$ for any $x\in\mathcal{C}$. Thus convexity of $g$ implies that for any $x\in\mathcal{C}$

[TABLE]

From assumptions (A.1) and (A.2), $f$ belongs to $\Gamma_{0}\left(\mathcal{H}_{p}\right)$ and is differentiable on an open set $\mathcal{C}_{0}$ that contains $\mathcal{C}\subset\mathrm{dom}(f)$ (see Definition 2.6). Thus the continuity set of $f$ contains $\mathcal{C}$, and it follows from [4, Corollary 8.30(ii)] that $\mathcal{C}\subset\mathrm{int}\left(\mathrm{dom}\left(f\right)\right)$. Consequently, arguing as in the proof of Lemma 3.2, we deduce that

[TABLE]

In turn, convexity entails that for any $x\in\mathcal{C}$

[TABLE]

From assumption (A.5), we also have for any $x\in\mathcal{C}$

[TABLE]

Summing (4.31), (4.33) and (4.34), using (4.32) and assumption (A.4), we get

[TABLE]

From Lemma 4.11, we know that the sequence of dual variables $\left(\mu_{k}\right)_{k\in\mathbb{N}}$ is bounded which concludes the proof. ∎

Define $C_{k}\stackrel{{\scriptstyle\text{\rm\tiny def}}}{{=}}\frac{L_{k}}{2}d_{\mathcal{C}}^{2}+d_{\mathcal{C}}\left(D+M\|T\|+L_{h}+\|A\|\ \|{\mu}^{\star}\|\right)$, where $L_{k}$ is given in (4.7) and the constants $D$, $M$, and $L_{h}$ are as in Lemma 4.7. We then have the following lemma, in which we state the main energy estimation.

Lemma 4.13**.**

Suppose that assumptions (A.1)-(A.8) and (P.1)-(P.6) hold, with $\underline{M}\geq 1$. Consider the sequence of primal-dual iterates $\left((x_{k},\mu_{k})\right)_{k\in\mathbb{N}}$ generated by Algorithm 1 and $\left({x}^{\star},{\mu}^{\star}\right)$ a saddle-point point of the Lagrangian as in (3.5). Let

[TABLE]

Then, we have the following energy estimate

[TABLE]

• Proof.

Notice that the dual update $\mu_{k+1}=\mu_{k}+\theta_{k}\left(Ax_{k+1}-b\right)$ can be re-written as

[TABLE]

Then, from firm nonexpansiveness of the proximal mapping (see (2.3)),

[TABLE]

Notice that

[TABLE]

and that, by the definition of $x_{k+1}$ in the algorithm and by convexity of function $h$,

[TABLE]

Then,

[TABLE]

Now apply Lemma 4.6 at the points ${x}^{\star}$, $x_{k}$, and $\mu_{k}$ to affirm that

[TABLE]

From the latter, by the alternative definition of $s_{k}$ in the algorithm (see (3.3)), we obtain

[TABLE]

From Lemma 4.5, we have also that

[TABLE]

Recall that, from the algorithm, $x_{k+1}=x_{k}+\gamma_{k}\left(s_{k}-x_{k}\right)$. Then,

[TABLE]

where in the last inequality we used (4.39). Using the latter in (4.38), we obtain

[TABLE]

Notice also that, from the definitions of $\mathcal{L}_{k}\left(x_{k+1},\cdot\right)$ and $\mu_{k+1}$ as $\mu_{k+1}=\mu_{k}+\theta_{k}\left(Ax_{k+1}-b\right)$,

[TABLE]

So, from the latter and (4.40),

[TABLE]

Now recall that, by assumption (P.6), $\theta_{k}=\gamma_{k}/c$. Multiply (4.37) by $c$ and sum with the latter, to obtain

[TABLE]

The previous inequality can be re-written, by trivial manipulations, as

[TABLE]

By (P.5) and (P.6), and the assumption that $\underline{M}\geq 1$, we have $\theta_{k+1}\leq\underline{M}^{-1}\theta_{k}\leq\theta_{k}$. In view of (P.3), we also have $\beta_{k+1}\leq\beta_{k}$ by (P.3). In particular, $g^{\beta_{k}}\leq g^{\beta_{k+1}}\leq g$. Now, by Proposition 2.1(iv) and the definition of the constant $M$ in (3.7), we are able to estimate the quantity

[TABLE]

Then,

[TABLE]

Recall that, by assumption (A.3), $\mathcal{C}$ is convex and bounded and that, by the update $x_{k+1}=x_{k}+\gamma_{k}\left(s_{k}-x_{k}\right)$ with $s_{k}\in\mathcal{C}$ and $\gamma_{k}\in]0,1]$ by (P.1), $x_{k}$ always belongs to $\mathcal{C}$. From the assumptions, the functions $f,h$ and $g\circ T$ are bounded on $\mathcal{C}$ and, from the algorithm and convexity, $\left(x_{k}\right)_{k\in\mathbb{N}}\subset\mathcal{C}$. By Lemma 4.11, also the sequence $\left(\mu_{k}\right)_{k\in\mathbb{N}}$ is bounded. Then, recalling $\tilde{M}$ from Lemma 4.12, we can use the Cauchy-Schwartz and the triangular inequality to affirm that

[TABLE]

Recall the definition of $r_{k}$ in (4.35). Coming back to (4.41) and using both (4.42) and (4.43), we obtain

[TABLE]

Recall that, by feasibility of ${x}^{\star}$, $\mathcal{L}\left({x}^{\star},\mu_{k}\right)=\mathcal{L}\left({x}^{\star},{\mu}^{\star}\right)$. Now compute

[TABLE]

where in the inequality we used the facts that $g^{\beta_{k}}\leq g$ and that, by Proposition 2.1(v) and (3.7),

[TABLE]

Then, using the latter in (4.44), we obtain

[TABLE]

We replace the term $\left[\mathcal{L}\left({x}^{\star},{\mu}^{\star}\right)-\mathcal{L}\left(x_{k+1},{\mu}^{\star}\right)\right]$ with $\left[\mathcal{L}\left({x}^{\star},{\mu}^{\star}\right)-\mathcal{L}\left(x_{k},{\mu}^{\star}\right)\right]+\left[\mathcal{L}\left(x_{k},{\mu}^{\star}\right)-\mathcal{L}\left(x_{k+1},{\mu}^{\star}\right)\right]$ and estimate using Lemma 4.7 to get the following,

[TABLE]

We conclude by trivial manipulations. ∎

We are now ready to prove Theorem 4.2.

• Proof.

Our starting point is the main energy estimate (4.36). Let us focus on its right-hand-side. Under assumption (P.7),

[TABLE]

where the right hand side is in $\ell^{1}_{+}$ by Lemma 4.9. Now remember that $C_{k}=\frac{L_{k}}{2}d_{\mathcal{C}}^{2}+d_{\mathcal{C}}\left(D+M\|T\|+L_{h}+\|A\|\ \|{\mu}^{\star}\|\right)$, where $L_{k}=\|T\|^{2}/\beta_{k}+\|A\|^{2}\rho_{k}$. Then we have

[TABLE]

Indeed, under assumption (P.1), the sequences $\left(\gamma_{k}\beta_{k}\right)_{k\in\mathbb{N}},\left(\zeta\left(\gamma_{k}\right)\right)_{k\in\mathbb{N}}$, and $\left(\gamma_{k}^{2}/\beta_{k}\right)_{k\in\mathbb{N}}$ belong to $\ell_{+}^{1}$. Moreover, we have by assumptions (P.3) and (P.4) that $\underline{\rho}\gamma_{k}^{2}\leq\rho_{k}\gamma_{k}^{2}\leq\beta_{0}\overline{\rho}\gamma_{k}^{2}/\beta_{k}$, whence we get that $\left(\rho_{k}\gamma_{k}^{2}\right)_{k\in\mathbb{N}}\in\ell_{+}^{1}$ and $\left(\gamma_{k}^{2}\right)_{k\in\mathbb{N}}\in\ell_{+}^{1}$ after invoking assumption (P.1). Thus all terms on the right hand side are summable. Let

[TABLE]

So far, we have shown that

[TABLE]

where $r_{k}$ is bounded from below, and $\left(z_{k}\right)_{k\in\mathbb{N}}\in\ell_{+}^{1}$. The rest of the proof consists of invoking properly Lemma 2.14.

1. (i)

In order to use Lemma 2.14(ii), we need to show that for some positive constant $\alpha$,

[TABLE]

Notice that the term $\mathcal{L}\left(x_{k},{\mu}^{\star}\right)-\mathcal{L}\left({x}^{\star},{\mu}^{\star}\right)$ is proportional to $\gamma_{k}$ by Lemma 4.7. For the second term of $w_{k}$, we have by Lemma 4.8 that $\frac{\rho_{k}}{2}\left\|Ax_{k}-b\right\|^{2}-\frac{\rho_{k+1}}{2}\left\|Ax_{k+1}-b\right\|^{2}$ is proportional to $\gamma_{k}$. The desired claim then follows from Lemma 2.14(ii). 2. (ii)

By [4, Lemma 2.37], we can assert that $\left(x_{k}\right)_{k\in\mathbb{N}}$ possesses a weakly convergent subsequence, say $\left(x_{k_{j}}\right)_{j\in\mathbb{N}}$, with cluster point $\bar{x}\in\mathcal{C}$. Since $\left\|A\cdot-b\right\|\in\Gamma_{0}(\mathcal{H}_{p})$ and in view of [4, Theorem 9.1], we have

[TABLE]

where we used lower semicontinuity of the norm and Theorem 4.2. Thus $A\bar{x}=0$, meaning that $\bar{x}$ is a feasible point of ($\mathrsfs{P}$). In turn, $\mathcal{L}\left(\bar{x},{\mu}^{\star}\right)=\Phi(\bar{x})$. The function $\mathcal{L}\left(\cdot,{\mu}^{\star}\right)$ is lower semicontinuous by (A.1) and (A.6). Thus, using [4, Theorem 9.1] and by virtue of claim (i), we have

[TABLE]

for all $x\in\mathcal{H}_{p}$, and in particular for all $x\in A^{-1}(b)$. Thus, for every $x\in A^{-1}(b)$, we deduce that

[TABLE]

meaning that $\bar{x}$ is a solution for problem ($\mathrsfs{P}$).

Meanwhile, as the sequence $\left(\mu_{k}\right)_{k\in\mathbb{N}}$ is bounded by Lemma 4.11, we can again invoke [4, Lemma 2.37] to extract a weakly convergent subsequence $\left(\mu_{k_{j}}\right)_{j\in\mathbb{N}}$ with cluster point $\bar{\mu}$. By Fermat’s rule ([4, Theorem 16.2]), the weak sequential cluster point $\bar{\mu}$ is a solution to ($\mathrsfs{D}$) if and only if

[TABLE]

Since the proximal operator is the resolvent of the subdifferential, it follows that (4.24) is equivalent to

[TABLE]

By Lemma 4.9 it follows that $A\tilde{x}_{k}$ converges strongly to $b$ and, combined with (4.25), thus $\nabla\varphi_{k_{j}}\left(\mu_{k_{j}}\right)$ converges strongly to [math]. On the other hand, $\mu_{k_{j}}-\rho_{k_{j}}\nabla\varphi_{k_{j}}\left(\mu_{k_{j}}\right)$ converges weakly to $\bar{\mu}$. We now argue that we can pass to the limit in (4.46) by showing sequential closedness.

When $g\equiv 0$, we have, for all $j\in\mathbb{N}$, $\Phi_{k_{j}}\equiv f+h$ and the rest of the argument relies on sequential closedness of the graph of the subdifferential of $\Phi^{*}\circ(-A^{*})\in\Gamma_{0}(\mathcal{H}_{d})$ in the weak-strong topology. For the general case, our argument will rely on the fundamental concept of Mosco convergence of functions, which is epigraphical convergence for both the weak and strong topology (see [9] and [2, Definition 3.7]).

By Proposition 2.1(v) and assumptions (A.1)-(A.2), $\left(\Phi_{k_{j}}\right)_{j\in\mathbb{N}}$ is an increasing sequence of functions in $\Gamma_{0}\left(\mathcal{H}_{d}\right)$. It follows from [2, Theorem 3.20(i)] that $\Phi_{k_{j}}$ Mosco-converges to $\sup_{j\in\mathbb{N}}\Phi_{k_{j}}=\sup_{j\in\mathbb{N}}f+g^{\beta_{k_{j}}}\circ T+h=f+g\circ T+h=\Phi$ since $\beta_{k_{j}}\to 0$ by (P.3). Bicontinuity of the Legendre-Fenchel conjugation for the Mosco convergence (see [2, Theorem 3.18]) entails that $\Phi_{k_{j}}^{*}\circ-(A^{*})$ Mosco-converges to $(f+g\circ T+h)^{*}\circ(-A^{*})=\Phi^{*}\circ(-A^{*})$. This implies, via [2, Theorem 3.66], that $\partial\Phi_{k_{j}}^{*}\circ(-A^{*})$ graph-converges to $\partial\Phi^{*}\circ(-A^{*})$, and [2, Proposition 3.59] shows that $\left(\partial\Phi_{k_{j}}\circ(-A^{*})\right)_{j\in\mathbb{N}}$ is sequentially closed for graph-convergence in the weak-strong topology on $\mathcal{H}_{d}$, i.e., for any sequence $(v_{k_{j}},\eta_{k_{j}})$ in the graph of $\partial\Phi_{k_{j}}^{*}\circ(-A^{*})$ such that $v_{k_{j}}$ converges weakly to $\bar{v}$ and $\eta_{k_{j}}$ converges strongly to $\bar{\eta}$, we have $\bar{\eta}\in\partial\Phi^{*}\circ(-A^{*})(\bar{v})$. Taking $v_{k_{j}}=\nabla\varphi_{k_{j}}\left(\mu_{k_{j}}\right)-b$ and $\eta_{k_{j}}=\mu_{k_{j}}-\rho_{k_{j}}\nabla\varphi_{k_{j}}\left(\mu_{k_{j}}\right)$, we conclude that

[TABLE]

i.e., $\bar{\mu}$ is a solution of the dual problem ($\mathrsfs{D}$).

Recall $r_{k}$ from (4.35) which verifies (4.45). From Lemma 2.14(i), $\left(r_{k}\right)_{k\in\mathbb{N}}$ is convergent. By (P.1) and (P.3), $\gamma_{k}$ and $\beta_{k}$ both converge to [math]. We also have that

[TABLE]

We have from Theorem 4.1(i) that $\frac{\rho_{k}}{2}\left\|Ax_{k}-b\right\|^{2}\to 0$. In turn, $\left\langle{\mu}^{\star}-\mu_{k},Ax_{k}-b\right\rangle\to 0$ since $\left(\mu_{k}\right)_{k\in\mathbb{N}}$ is bounded (Lemma 4.11). We also have $\mathcal{L}(x_{k},{\mu}^{\star})\to\mathcal{L}({x}^{\star},{\mu}^{\star})$ by claim (i) above. By Proposition 2.1(v) and (3.7), we get that

[TABLE]

Passing to the limit and in view of (P.3), we conclude that $g(Tx_{k})-g^{\beta_{k}}(Tx_{k})\to 0$. Altogether, this shows that $\mathcal{L}_{k}\left(x_{k},\mu_{k}\right)\to\mathcal{L}({x}^{\star},{\mu}^{\star})$. In turn, we conclude that the limit

[TABLE]

exists. Since ${\mu}^{\star}$ was an arbitray optimal dual point, and we have shown above that each subsequence of $\left(\mu_{k}\right)_{k\in\mathbb{N}}$ converges weakly to an optimal dual point, we are in position to invoke Opial’s lemma [29] to conclude that the whole dual multiplier sequence weakly converges to a solution of the dual problem. 3. (iii)

Recalling that $\left(\gamma_{k}\right)_{k\in\mathbb{N}}\not\in\ell^{1}_{+}$ (see assumption (P.2)), the rates in (4.4) follow by applying Lemma 2.14(iii)-(iv) to (4.45). Notice that both terms in $w_{k}$ are positive and that $\rho_{k}\geq\underline{\rho}>0$ (see again assumption (P.4)). Therefore we have that, for the same subsequence $\left(x_{k_{j}}\right)_{j\in\mathbb{N}}$, (4.6) holds. 4. (iv)

The ergodic rate (4.2) follows by applying the Jensen’s inequality to the convex function $\mathcal{L}\left(\cdot,{\mu}^{\star}\right)$.

∎

5 Applications

5.1 Sum of several nonsmooth functions

In this section we explore the applications of Algorithm 1 to splitting in composite optimization problems, where we allow the presence of more than one nonsmooth function $g$ or $h$ in the objective:

[TABLE]

First, we denote the product space by $\boldsymbol{\mathcal{H}}_{p}\stackrel{{\scriptstyle\text{\rm\tiny def}}}{{=}}\mathcal{H}_{p}^{n}$ endowed with the scalar product $\boldsymbol{\langle\kern-5.0pt\langle}\boldsymbol{x},\boldsymbol{y}\boldsymbol{\rangle\kern-5.0pt\rangle}=\frac{1}{n}\sum_{i=1}^{n}\left\langle x^{\left(i\right)},y^{\left(i\right)}\right\rangle$, where $\boldsymbol{x}$ and $\boldsymbol{y}$ are vectors in $\boldsymbol{\mathcal{H}}_{p}$ with $\boldsymbol{x}\stackrel{{\scriptstyle\text{\rm\tiny def}}}{{=}}\begin{pmatrix}x^{\left(1\right)},\dots,x^{\left(n\right)}\end{pmatrix}^{\top}$. We define also $\mathcal{V}$ as the diagonal subspace of $\boldsymbol{\mathcal{H}}_{p}$, i.e. $\mathcal{V}\stackrel{{\scriptstyle\text{\rm\tiny def}}}{{=}}\small\{\boldsymbol{x}\in\boldsymbol{\mathcal{H}}_{p}:x^{\left(1\right)}=\ldots=x^{\left(n\right)}\small\}$, $\mathcal{V}^{\perp}$ the orthogonal subspace to $\mathcal{V}$, and $\Pi_{\mathcal{V}},\Pi_{\mathcal{V}^{\perp}}$ the orthogonal projections onto $\mathcal{V}$, $\mathcal{V}^{\perp}$ - respectively. We finally introduce the (diagonal) linear operator $\boldsymbol{T}:\boldsymbol{\mathcal{H}}_{p}\to\boldsymbol{\mathcal{H}}_{p}$ defined by

[TABLE]

and the functions

[TABLE]

Then problem (5.1) is obviously equivalent to

[TABLE]

which fits in the setting of our main problem ($\mathrsfs{P}$). In order to make more clear the presentation, we separate the two cases of multiple $g$ and multiple $h$, that can be trivially combined. Moreover, we focus on the main case $h_{i}=\iota_{\mathcal{C}_{i}}$.

5.2 Sum of several simple functions over a compact set

Consider the following composite minimization problem,

[TABLE]

We can reformulate the problem in the product space $\boldsymbol{\mathcal{H}}_{p}$ using the above notation to get,

[TABLE]

Applying Algorithm 1 to this problem gives a completely separable scheme; we first compute the direction,

[TABLE]

which reduces to the following computation since $\boldsymbol{s}_{k}=\begin{pmatrix}s_{k}\\ \vdots\\ s_{k}\end{pmatrix}$ has identical components,

[TABLE]

The term $\nabla g_{i}^{\beta_{k}}$ has a closed form given in Proposition 2.1 which can be used to get the following formula for the direction,

[TABLE]

5.3 Minimizing over intersection of compact sets

A classical problem found in machine learning is to minimize a Lipschitz-smooth function $f$ over the intersection of convex, compact sets $\mathcal{C}_{i}$ in some real Hilbert space $\mathcal{H}$,

[TABLE]

where $h_{i}\equiv\iota_{\mathcal{C}_{i}}$. Reformulating the problem in the product space $\boldsymbol{\mathcal{H}}_{p}$ gives,

[TABLE]

Then, we can apply Algorithm 1 and compute the step direction

[TABLE]

which gives a separable scheme for each component of $\boldsymbol{s}_{k}=\begin{pmatrix}s_{k}^{\left(1\right)}\\ \vdots\\ s_{k}^{\left(n\right)}\end{pmatrix}$,

[TABLE]

6 Comparison

6.1 Conditional Gradient Framework

In [37] the following problem was analyzed in the finite-dimensional setting,

[TABLE]

where $f\in C^{1,1}\left(\mathbb{R}^{n}\right)\cap\Gamma_{0}\left(\mathbb{R}^{n}\right)$, $T\in\mathbb{R}^{d\times n}$ is a linear operator, $g\circ T\in\Gamma_{0}\left(\mathbb{R}^{n}\right)$, and $\mathcal{C}$ is a compact, convex subset of $\mathbb{R}^{n}$. They develop an algorithm which avoids projecting onto the set $\mathcal{C}$, instead utilizing a linear minimization oracle $\operatorname*{lmo}_{\mathcal{C}}\left(v\right)=\operatorname*{Argmin}\limits_{x\in\mathcal{C}}\left\langle x,v\right\rangle$, and replaces the function $g\circ T$ with the smooth function $g^{\beta}_{k}\circ T$. They consider only functions $f$ which are Lipschitz-smooth and finite dimensional spaces, i.e. $\mathbb{R}^{n}$, compared to CGALP which weakens the assumptions on $f$ to be differentiable and $\left(F,\zeta\right)$-smooth (see Definition 2.6) with an arbitrary real Hilbert space $\mathcal{H}_{p}$ (possibly infinite dimensional). Furthermore, the analysis in [37] is restricted to the parameter choices $\gamma_{k}=\frac{2}{k+1}$ and $\beta_{k}=\frac{\beta_{0}}{\sqrt{k+1}}$ exclusively, although they do include a section in which they consider two variants of an inexact linear minimization oracle: one with additive noise and one with multiplicative noise. In contrast, the results we present in Section 3 show optimality and feasibility for a wider choice for both the sequence of stepsizes $\left(\gamma_{k}\right)_{k\in\mathbb{N}}$ and the sequence of smoothing parameters $\left(\beta_{k}\right)_{k\in\mathbb{N}}$, although we only consider exact linear perturbation oracles of the form $\operatorname*{Argmin}\limits_{s\in\mathcal{H}_{p}}\left\{h\left(s\right)+\left\langle x,s\right\rangle\right\}$. Finally, for solving (6.1) with an exact linear minimization oracle, our algorithm encompasses the algorithm in [37] by choosing $h\left(x\right)=\iota_{\mathcal{C}}\left(x\right)$, $A\equiv 0$, and restricting $f$ to be in $C^{1,1}\left(\mathcal{H}\right)$ with $\mathcal{H}=\mathbb{R}^{n}$.

In [37, Section 5] there is a discussion on splitting and affine constraints using the conditonal gradient framework presented. In this setting, i.e. assuming exact oracles, the primary difference between CGALP and the conditional gradient framework is the approach each algorithm takes to handle affine constraints. In CGALP , the augmented Lagrangian formulation is used to account for the affine constraints, introducing a dual variable $\mu$ and both a linear and quadratic term for the constraint $Ax-b=0$. In contrast, in [37] the affine constraint is treated the same as the nonsmooth term $g\circ T$ and thus handled by quadratic penalization/smoothing alone. The consequence of smoothing for the affine constraint $Ax=b$ comes from calculating the gradient of the squared-distance to the constraint. This will involve solving a least squares problem at each iteration which can be computationally expensive. Our algorithm does not need to solve such a linear system.

The difference in the approaches is highlighted when both methods are applied to problem presented in Section 5.3 with $n=2$ since this problem necessitates an affine constraint $\Pi_{\mathcal{V}^{\perp}}\boldsymbol{x}=0$ for splitting. According to [37, Section 5], we reformulate the problem to be

[TABLE]

Note that the inclusion of the function $\iota_{\left\{x^{\left(1\right)}\right\}}\left(x^{\left(2\right)}\right)$ in the objective is equivalent to the affine constraint $\Pi_{\mathcal{V}^{\perp}}\boldsymbol{x}=0$ in the $n=2$ case. Apply the conditional gradient framework on the variable $\left(x^{\left(1\right)},x^{\left(2\right)}\right)$ to get

[TABLE]

which leads to a separable scheme that can be computed component-wise,

[TABLE]

Compare the direction obtained in (6.2) to the one obtained in (5.4), the components of which we rewrite below for $n=2$,

[TABLE]

Due to affine constraint, the computation of the direction in (6.2) necessitates smoothing and, as a consequence, the parameter $\beta_{k}$, which is necessarily going to [math]. In CGALP , the introduction of the dual variable $\boldsymbol{\mu}_{k}$ in place of smoothing the affine constraint avoids the parameter $\beta_{k}$. Instead, we have the parameter $\rho_{k}$ but $\rho_{k}$ can be picked to be constant without issue.

6.2 FW-AL Algorithm

In [16] the following problem was analyzed,

[TABLE]

using a combination of the Frank-Wolfe algorithm with the augmented Lagrangian to account for the constraint $Ax=0$. The function $f$ is assumed to be Lipschitz-smooth, in contrast to our approach. The perspective used in their paper is to modify the classic ADMM algorithm, replacing the marginal minimization with respect to the primal variable by a Frank-Wolfe step instead, although their analysis is not restricted only to Frank-Wolfe steps. Indeed, in all the scenarios where one can apply FW-AL using a Frank-Wolfe step our algorithm encompasses FW-AL as a special case, discussed in Section 5.3. The primary differences between CGALP and FW-AL are in the convergence results and the generality of CGALP . The results in [16] prove convergence of the objective in the case where the sets $\mathcal{C}_{i}$ are polytopes and convergence of the iterates in the case where the sets $\mathcal{C}_{i}$ are polytopes and $f$ is strongly convex, but they do not prove convergence of the objective, weak convergence of the dual variable, or asymptotic feasibility of the iterates in the general case where each $\mathcal{C}_{i}$ is a compact, convex set. Instead, they prove two theorems which imply subsequential convergence of the objective and subsequential asymptotic feasibility in the general case and subsequential convergence of the iterates to the optimum in the strongly convex case in [16, Theorem 2] and [16, Corollary 2] respectively. Unfortunately, each of these results is obtained separately and so the subsequences that produce each result are not guaranteed to coincide with one another.

Interestingly, the results they obtain are not unique to Frank-Wolfe style algorithms as their analysis is from the perspective of a modified ADMM algorithm; they only require that the algorithm used to replace the marginal minimization on the primal variable in ADMM produces sublinear decrease in the objective. Finally, they do not provide conditions for the dual multiplier sequence, $\mu_{k}$ in our notation, to be bounded as they discuss in their analysis of issues with similar proofs, e.g. in GDMM. This is a crucial issue as the constants in their bounds depend on the norm of these dual multipliers.

7 Numerical Experiments

In this section we present some numerical experiments comparing the performance of Algorithm 1 and a proximal algorithm applied to splitting in composite optimization problems.

7.1 Projection problem

First, we consider a simple projection problem,

[TABLE]

where $y\in\mathbb{R}^{2}$ is the vector to be projected and $A:\mathbb{R}^{2}\to\mathbb{R}^{2}$ is a rank-one matrix. To exclude trivial projections, we choose randomly $y\notin\mathbb{B}_{1}^{1}\cap\mathrm{ker}(A)$, where $\mathbb{B}_{1}^{1}$ is the unit $\ell^{1}$ ball centered at the origin. Then Problem (7.1) is nothing but Problem ($\mathrsfs{P}$) with $f\left(x\right)=\frac{1}{2}\left\|x-y\right\|_{2}^{2}$, $g\equiv 0$, $h\equiv\iota_{\mathbb{B}_{1}^{1}}$ and $\mathcal{C}=\mathbb{B}_{1}^{1}$.

The assumptions mentioned previously, i.e. (A.1)-(A.8), all hold in this finite-dimensional case as $f$, $g$, and $h$ are all in $\Gamma_{0}\left(\mathbb{R}^{2}\right)$, $f$ is Lipschitz-smooth, $h$ is the indicator function for a compact convex set, $g$ has full domain and $0\in\mathrm{ker}(A)\cap\mathrm{int}(\mathcal{C})$. Regarding the parameters and the associated assumptions, we choose $\gamma_{k}$ according to Example 3.4 with $(a,b)\in\{(0,0),(0,1/3-0.01),(1,1/3-0.01)\}$, $\theta_{k}=\gamma_{k}$, and $\rho=2^{2-b}+1$. The ergodic convergence profiles of the Lagrangian are displayed in Figure 1 along with the theoretical rates (see Theorem 4.2 and Example 4.4). The observed rates agree with the predicted ones of $O\left(\frac{1}{\log(k+2)}\right)$, $O\left(\frac{1}{(k+2)^{b}}\right)$ and $o\left(\frac{1}{(k+2)^{b}}\right)$ for the respective choices of $(a,b)$.

7.2 Matrix completion problem

We also consider the following, more complicated matrix completion problem,

[TABLE]

where $\delta_{1}$ and $\delta_{2}$ are positive constants, $\Omega:\mathbb{R}^{N\times N}\to\mathbb{R}^{p}$ is a masking operator, $y\in\mathbb{R}^{p}$ is a vector of observations, and $\left\|\cdot\right\|_{*}$ and $\left\|\cdot\right\|_{1}$ are respectively the nuclear and $\ell^{1}$ norms. The mask operator $\Omega$ is generated randomly by specifying a sampling density, in our case $0.8$. We generate the vector $y$ randomly in the following way. We first generate a sparse vector $\tilde{y}\in\mathbb{R}^{N}$ with $N/5$ non-zero entries independently uniformly distributed in $[-1,1]$. We take the exterior product $\tilde{y}\tilde{y}^{\top}=X_{0}$ to get a rank-1 sparse matrix which we then mask to get $\Omega X_{0}$. The radii of the contraints in (7.2) are chosen according to the nuclear norm and $\ell^{1}$ norm of $X_{0}$, $\delta_{1}=\frac{\left\|X_{0}\right\|_{*}}{2}$ and $\delta_{2}=\frac{\left\|X_{0}\right\|_{1}}{2}$.

7.2.1 CGALP

Problem (7.2) is a special instance of (5.1) with $n=2$, $f\equiv 0$, $g_{i}=\left\|\cdot-y\right\|_{1}/2$, $T_{i}=\Omega$, $h_{1}=\iota_{\mathbb{B}_{*}^{\delta_{1}}}$, $h_{2}=\iota_{\mathbb{B}_{1}^{\delta_{2}}}$, where $\mathbb{B}_{*}^{\delta_{1}}$ and $\mathbb{B}_{1}^{\delta_{2}}$ are the nuclear and $\ell^{1}$ balls of radii $\delta_{1}$ and $\delta_{2}$. We then follow the same steps as in Section 5.1. Let $\mathcal{H}_{p}=\mathbb{R}^{N\times N}$, $\boldsymbol{\mathcal{H}}_{p}=\mathcal{H}_{p}^{2}$, $\boldsymbol{X}=\begin{pmatrix}X^{\left(1\right)}\\ X^{\left(2\right)}\end{pmatrix}\in\boldsymbol{\mathcal{H}}_{p}$. We then have $G\left(\Omega\boldsymbol{X}\right)=\frac{1}{2}\left(\left\|\Omega X^{\left(1\right)}-y\right\|_{1}+\left\|\Omega X^{\left(2\right)}-y\right\|_{1}\right)$, and $H(\boldsymbol{X})=\iota_{\mathbb{B}_{*}^{\delta_{1}}}(X^{\left(1\right)})+\iota_{\mathbb{B}_{1}^{\delta_{2}}}(X^{\left(2\right)})$. Then problem (7.2) is obviously equivalent to

[TABLE]

which is a special case of (5.2) with $F\equiv 0$. It is immediate to check that our assumptions (A.1)-(A.8) hold. Indeed, all functions are in $\Gamma_{0}(\mathcal{H}_{p})$ and $F\equiv 0$, and thus (A.1) and (A.2) are verified. $\mathcal{C}=\mathbb{B}_{*}^{\delta_{1}}\times\mathbb{B}_{1}^{\delta_{2}}$ which is a non-mepty convex compact set. We also have $\Omega\mathcal{C}\subset\mathrm{dom}(\partial G)=\mathbb{R}^{p}\times\mathbb{R}^{p}$, and for any $\boldsymbol{z}\in\mathbb{R}^{p}\times\mathbb{R}^{p}$, $\partial G(\boldsymbol{z})\subset\mathbb{B}_{\infty}^{1/2}\times\mathbb{B}_{\infty}^{1/2}$ and thus (A.4) is verified. (A.5) also holds with $L_{h}=0$. $\mathcal{V}$ is closed as we are in finite dimension, and thus (A.7) is fulfilled. We also have, since $\mathrm{dom}(G\circ\Omega)=\boldsymbol{\mathcal{H}}_{p}$,

[TABLE]

which shows that (A.8) is verified. The latter is nothing but the condition in [4, Fact 15.25(i)]. It then follows from the discussion in Remark 3.1(iv) that (A.6) holds true.

We use Algorithm 1 by choosing the sequence of parameters $\gamma_{k}=\frac{1}{k+1}$, $\beta_{k}=\frac{1}{\sqrt{k+1}}$, $\theta_{k}=\gamma_{k}$, and $\rho_{k}\equiv 15$, which verify all our assumptions (P.1)-(P.7) in view of Example 3.4. Our choice of $\gamma_{k}$ is the most common in the literature, and it can be improved according to our discussion in the previous section.

Finding the direction $\boldsymbol{S}_{k}$ by solving the linear minimization oracle is a separable problem, and thus each component is given by,

[TABLE]

Because of the structure of the sets $\mathbb{B}_{\left\|\cdot\right\|_{*}}^{\delta_{1}}$ and $\mathbb{B}_{\left\|\cdot\right\|_{1}}^{\delta_{2}}$, finding the first component of $\boldsymbol{S}_{k}$ reduces to computing the leading right and left singular vectors of

[TABLE]

while finding the second component reduces to computing the largest entry of

[TABLE]

over all the entries $\left(i,j\right)$. The dual variable update is given by,

[TABLE]

7.2.2 GFB

Let $\mathcal{H}_{p}=\mathbb{R}^{N\times N}$, $\boldsymbol{\mathcal{H}}_{p}=\mathcal{H}_{p}^{3}$, $\boldsymbol{W}=\begin{pmatrix}W^{\left(1\right)}\\ W^{\left(2\right)}\\ W^{\left(3\right)}\end{pmatrix}\in\boldsymbol{\mathcal{H}}_{p}$, $Q\left(\boldsymbol{W}\right)=\left\|\Omega W^{\left(1\right)}-y\right\|_{1}+\iota_{\mathbb{B}_{\left\|\cdot\right\|_{*}}^{\delta_{1}}}\left(W^{\left(2\right)}\right)+\iota_{\mathbb{B}_{\left\|\cdot\right\|_{1}}^{\delta_{2}}}\left(W^{\left(3\right)}\right)$. Then we reformulate problem (7.2) as

[TABLE]

which fits the framework to apply the GFB algorithm proposed in [32] (in fact Douglas-Rachford since the smooth part vanishes).

The algorithm has three steps, each of which is separable in the components. We choose the step sizes $\lambda_{k}=\gamma=1$ in the GFB to get,

[TABLE]

We know from [32] that $\boldsymbol{Z}_{k}$ converges to $\boldsymbol{Z}^{\star}$, and $\boldsymbol{W}_{k}$ and $\boldsymbol{U}_{k}$ both converge to $\boldsymbol{W}^{\star}=\Pi_{\mathcal{V}}(\boldsymbol{Z}^{\star})=(X^{\star},X^{\star},X^{\star})$, where $X^{\star}$ is a minimizer of (7.2).

7.2.3 Results

We compare the performance of CGALP with GFB for varying dimension, $N$, using their respective ergodic convergence criteria. For CGALP this is the quantity $\mathcal{L}\left(\bar{\boldsymbol{X}}_{k},\mu^{*}\right)-\mathcal{L}\left(\boldsymbol{X}^{\star},\boldsymbol{\mu}^{\star}\right)$ where $\bar{\boldsymbol{X}}_{k}=\sum\limits_{i=0}^{k}\gamma_{i}\boldsymbol{X}_{i}/\Gamma_{k}$. Meanwhile, for GFB, we know from [25] that the Bregman divergence $D_{Q}^{\boldsymbol{v}^{\star}}\left(\bar{\boldsymbol{U}}_{k}\right)=Q(\bar{\boldsymbol{U}}_{k})-Q(\boldsymbol{W}^{\star})-\left\langle\boldsymbol{v}^{\star},\bar{\boldsymbol{U}}_{k}-\boldsymbol{W}^{\star}\right\rangle$, with $\bar{\boldsymbol{U}}_{k}=\sum\limits_{i=0}^{k}\boldsymbol{U}_{i}/(k+1)$ and $\boldsymbol{v}^{\star}=(\boldsymbol{W}^{\star}-\boldsymbol{Z}^{\star})/\gamma$, converges at the rate $O(1/(k+1))$. To compute the convergence criteria, we first run each algorithm for $10^{5}$ iterations to approximate the optimal variables ($\boldsymbol{X}^{\star}$ and $\boldsymbol{\mu}^{\star}$ for CGALP , and $\boldsymbol{Z}^{\star}$ and $\boldsymbol{W}^{\star}$ for GFB). Then, we run each algorithm again for $10^{5}$ iterations, this time recording the convergence criteria at each iteration. The results are displayed in Figure 2.

It can be observed that our theoretically predicted rate (which is $O\left(1/\log(k+2)\right)$ for CGALP according to Theorem 4.2 and Example 4.4) is in close agreement with the observed one. On the other hand, as is very well-known, employing a proximal step for the nuclear ball constraint will necessitate to compute an SVD which is much more time consuming than computing the linear minimization oracle for large $N$. For this reason, even though the rates of convergence guaranteed for CGALP are slower than for GFB, one can expect CGALP to be a more time computationally efficient algorithm for large $N$.

Acknowledgements

ASF was supported by the ERC Consolidated grant NORIA. JF was partly supported by Institut Universitaire de France. CM was supported by Project MONOMADS funded by Conseil Régional de Normandie. We would like to warmly thank Gabriel Peyré for his support and very fruitful and inspiring discussions.