Parallel Random Block-Coordinate Forward-Backward Algorithm: A Unified Convergence Analysis

TL;DR

This paper introduces a unified convergence analysis for a parallel, random block-coordinate forward-backward algorithm, demonstrating its convergence properties and rates under various conditions.

Contribution

It provides a comprehensive convergence analysis for a flexible parallel block-coordinate algorithm with arbitrary update probabilities and stepsizes.

Findings

Almost sure weak convergence in convex and infinite-dimensional settings

O(1/n) convergence rate for mean function values

Linear convergence under strong convexity and error bounds

Abstract

We study the block-coordinate forward-backward algorithm in which the blocks are updated in a random and possibly parallel manner, according to arbitrary probabilities. The algorithm allows different stepsizes along the block-coordinates to fully exploit the smoothness properties of the objective function. In the convex case and in an infinite dimensional setting, we establish almost sure weak convergence of the iterates and the asymptotic rate o(1/n) for the mean of the function values. We derive linear rates under strong convexity and error bound conditions. Our analysis is based on an abstract convergence principle for stochastic descent algorithms which allows to extend and simplify existing results.

Equations

$$\underset{\begin{array}{c}\mathsf{x}\in \mathsf{H}\end{array}}{\mathrm{minimize}}\mathsf{f}(\mathsf{x})+\mathsf{h}(\mathsf{x}),\mathsf{h}(\mathsf{x})=\sum _{i=1}^m{\mathsf{h}}_i({\mathsf{x}}_i),$$

$$\mathsf{H}=\underset{i=1}{\overset{m}{⨁}}{\mathsf{H}}_i,(\forall \mathsf{x}=({\mathsf{x}}_i{)}_{1\le i\le m},\mathsf{y}=({\mathsf{y}}_i{)}_{1\le i\le m}\in \mathsf{H})⟨\mathsf{x},\mathsf{y}⟩=\sum _{i=1}^m⟨{\mathsf{x}}_i,{\mathsf{y}}_i⟩,$$

$$\begin{array}[]{l}\text{for}\;n=0,1,\ldots\\ \left\lfloor\begin{array}[]{l}\text{for}\;i=1,\dots,m\\[3.01385pt] \left\lfloor\begin{array}[]{l}x^{n+1}_{i}=x^{n}_{i}+\varepsilon^{n}_{i}\big{(}\text{\sf prox}_{\gamma_{i}{\mathsf{h}}_{i}}\big{(}x^{n}_{i}-\gamma_{i}\nabla_{i}{\mathsf{f}}(x^{n})\big{)}-x^{n}_{i}\big{)}.\end{array}\right.\end{array}\right.\end{array}$$

$$(\forall \mathsf{x}\in \mathsf{H})(\forall \mathsf{v}\in \mathsf{H})\mathsf{f}(\mathsf{x}+\mathsf{v})\le \mathsf{f}(\mathsf{x})+⟨\nabla \mathsf{f}(\mathsf{x}),\mathsf{v}⟩+\frac{L}{2}\Vert \mathsf{v}{\Vert }^2.$$

$$(\forall i=1,\dots ,m)(\forall \mathsf{x}\in \mathsf{H})(\forall {\mathsf{v}}_i\in {\mathsf{H}}_i)\mathsf{f}(\mathsf{x}+{\mathsf{J}}_i{\mathsf{v}}_i)\le \mathsf{f}(\mathsf{x})+⟨{\nabla }_i\mathsf{f}(\mathsf{x}),{\mathsf{v}}_i⟩+\frac{L_i}{2}\Vert {\mathsf{v}}_i{\Vert }^2,$$

$$(\forall i\in \{1,\dots ,m\}){\gamma }_i<\frac{2}{{\nu }_i}.$$

$$x^{n+1}=x^{n}+\sum_{i=1}^{m}\varepsilon_{i}^{n}\mathsf{J}_{i}\big{(}\text{\sf prox}_{\gamma_{i}{\mathsf{h}}_{i}}\big{(}x^{n}_{i}-\gamma_{i}\nabla_{i}{\mathsf{f}}(x^{n})\big{)}-x^{n}_{i}\big{)}.$$

$$(\forall \mathsf{x}\in \mathsf{H})(\forall \mathsf{y}\in \mathsf{H})⟨\mathsf{x},\mathsf{y}{⟩}_{\mathsf{W}}=⟨\mathsf{W}\mathsf{x},\mathsf{y}⟩=\sum _{i=1}^m{w}_i⟨{\mathsf{x}}_i,{\mathsf{y}}_i⟩,$$

$$\partial^{{\mathsf{W}}}{\varphi}\colon\mathsf{H}\to 2^{\mathsf{H}},\ \ \mathsf{x}\mapsto\partial^{{\mathsf{W}}}{\varphi}(\mathsf{x})=\big{\{}\mathsf{u}\in\mathsf{H}\,|\,(\forall\,\mathsf{y}\in\mathsf{H})\ {\varphi}(\mathsf{y})\geq{\varphi}(\mathsf{x})+\langle\mathsf{u},\mathsf{y}-\mathsf{x}\rangle_{{\mathsf{W}}}\big{\}}.$$

$${\text{prox}}_{\phi }^{\mathsf{W}}:\mathsf{H}\to \mathsf{H},{\text{prox}}_{\phi }^{\mathsf{W}}(\mathsf{x})=\underset{\mathsf{z}\in \mathsf{H}}{\mathrm{argmin}}\phi (\mathsf{z})+\frac{1}{2}\Vert \mathsf{x}-\mathsf{z}{\Vert }_{\mathsf{W}}^2.$$

$$\mathsf{f}(\mathsf{y})$$

$$(\forall \mathsf{v}\in \partial \mathsf{h}(\mathsf{x}))\mathsf{h}(\mathsf{y})$$

$$(\forall i\in [m]){\mathsf{h}}_i({\mathsf{y}}_i)\ge {\mathsf{h}}_i({\mathsf{x}}_i)+⟨{\mathsf{v}}_i,{\mathsf{y}}_i-{\mathsf{x}}_i⟩+\frac{{\sigma }_{{\mathsf{\Gamma }}^{-1}}}{2}\frac{1}{{\gamma }_i}\Vert {\mathsf{y}}_i-{\mathsf{x}}_i{\Vert }^2.$$

$${\mu }_{{\mathsf{\Gamma }}^{-1}}\le \underset{1\le i\le m}{\min }{\gamma }_i{\nu }_i\le \delta <2.$$

$$\mathsf{f}(\mathsf{x})+\mathsf{E}[⟨\nabla \mathsf{f}(\mathsf{x}),\epsilon \odot {\mathsf{J}}_i{\mathsf{v}}_i⟩]+\frac{{\mu }_{{\mathsf{\Gamma }}^{-1}}}{2}\frac{{\mathsf{p}}_i}{{\gamma }_i}\Vert {\mathsf{v}}_i{\Vert }^2$$

$$\le \mathsf{f}(\mathsf{x})+\mathsf{E}[⟨\nabla \mathsf{f}(\mathsf{x}),\epsilon \odot {\mathsf{J}}_i{\mathsf{v}}_i⟩]+\frac{1}{2}{\mathsf{p}}_i{\nu }_i\Vert {\mathsf{v}}_i{\Vert }^2.$$

$$(\forall\,\mathsf{x}\in\mathsf{H})\qquad{\mathsf{f}}(\mathsf{x})=\sum_{k=1}^{p}{\mathsf{g}}_{k}\bigg{(}\sum_{i=1}^{m}\mathsf{U}_{k,i}\mathsf{x}_{i}\bigg{)},$$

$$(\forall\,i\in[m])\quad\nu_{i}\geq\beta_{1,i}L_{i},\quad\text{where}\quad\beta_{1,i}:=\mathsf{E}\Big{[}\max_{1\leq k\leq p}\Big{(}\sum_{j\in I_{k}}\varepsilon_{j}\Big{)}\,\Big{|}\,\varepsilon_{i}=1\Big{]}.$$

$$(\forall\,i\in[m])\quad\nu_{i}\geq\beta_{2}L_{i},\quad\text{where}\quad\beta_{2}:=\operatorname*{\text{\rm ess\>sup}}\Big{(}\max_{1\leq k\leq p}\Big{(}\sum_{j\in I_{k}}\varepsilon_{j}\Big{)}\Big{)}.$$

$$(\forall\,i\in[m])\quad\nu_{i}\geq\tilde{L}_{i}:=\Big{\lVert}\sum_{k=1}^{p}L^{(k)}\mathsf{U}_{k,i}^{\top}\mathsf{U}_{k,i}\Big{\rVert}.$$

$$\beta_{1,i}:=\sum_{t=1}^{\eta}t\max_{\begin{subarray}{c}1\leq k\leq p\\[0.90417pt] i\in I_{k}\end{subarray}}\mathsf{P}\Big{(}\sum_{j\in I_{k}}\varepsilon_{j}=t\,\big{|}\,\varepsilon_{i}=1\Big{)}.$$

$$\displaystyle\tau_{\max}:=\operatorname*{\text{\rm ess\>sup}}\Big{(}\sum_{i=1}^{m}\varepsilon_{i}\Big{)}$$

$$(\forall\,k\in[p])\quad\sum_{i\in I_{k}}\varepsilon_{i}\leq\min\Big{\{}\mathrm{card}(I_{k}),\sum_{i=1}^{m}\varepsilon_{i}\Big{\}}\leq\min\Big{\{}\eta,\sum_{i=1}^{m}\varepsilon_{i}\Big{\}}\leq\min\{\eta,\tau_{\max}\}.$$

$$\beta_{1,i}=\beta_{1}:=1+\frac{\eta-1}{m-1}\bigg{(}\frac{\mathsf{E}\big{[}\big{(}\sum_{i=1}^{m}\varepsilon_{i}\big{)}^{2}\big{]}}{\mathsf{E}[\sum_{i=1}^{m}\varepsilon_{i}]}-1\bigg{)}.$$

$${\beta }_{1,i}={\beta }_1:=1+\frac{(\eta -1)(\tau -1)}{m-1}.$$

$${\mathsf{\Gamma}}^{-1}=\bigoplus_{i=1}^{m}\frac{1}{\gamma_{i}}\mathsf{Id}_{i},\quad(w_{i})_{1\leq i\leq m}=\Big{(}\frac{1}{\gamma_{i}\mathsf{p}_{i}}\Big{)}_{1\leq i\leq m},\quad{\mathsf{W}}=\bigoplus_{i=1}^{m}w_{i}\mathsf{Id}_{i},$$

$$\bar{x}^{n+1}=\big{(}\text{\sf prox}_{\gamma_{i}{\mathsf{h}}_{i}}(x_{i}^{n}-\gamma_{i}\nabla_{i}{\mathsf{f}}(x^{n}))\big{)}_{1\leq i\leq m},\quad{\Delta}^{n}=x^{n}-\bar{x}^{n+1}.$$

$$\bar{x}^{n+1}=\text{\sf prox}_{{{\mathsf{h}}}}^{{\mathsf{\Gamma}}^{-1}}\big{(}x^{n}-\nabla^{{\mathsf{\Gamma}}^{-1}}{\mathsf{f}}(x^{n})\big{)},\quad x^{n+1}=x^{n}+\varepsilon^{n}\odot(\bar{x}^{n+1}-x^{n}),$$

$$\bar{x}^{n+1}_{i}=\text{\sf prox}_{\gamma_{i}{\mathsf{h}}_{i}}\big{(}x^{n}_{i}-\gamma_{i}\nabla_{i}{\mathsf{f}}(x^{n})\big{)}=x^{n+1}_{i},\quad\Delta^{n}_{i}=x^{n}_{i}-x^{n+1}_{i}.$$

$$\mathsf{E}[\Vert x^{n+1}-\mathsf{x}{\Vert }^2|{\mathfrak{X}}_n]\le \Vert x^n-\mathsf{x}{\Vert }^2+a\mathsf{E}[\mathsf{\Phi }(\mathsf{x})-\mathsf{\Phi }(x^{n+1})|{\mathfrak{X}}_n]+{\xi }_n\mathsf{P}\text{-a.s.}$$

$$(\forall\,n\in\mathbb{N},n\geq 1)\quad\mathsf{E}[{\mathsf{\Phi}}(x^{n})]-{\mathsf{\Phi}}_{*}\leq\bigg{[}\frac{\mathrm{dist}^{2}(\mathsf{x}^{0},\mathsf{S}_{*})}{a}+\frac{b}{a}\bigg{]}\frac{1}{n}.$$

Full text

Keywords. Convex optimization, parallel algorithms, random block-coordinate descent, arbitrary sampling, error bounds, stochastic quasi-Fejér sequences, forward-backward algorithm, convergence rates.

AMS Mathematics Subject Classification: 65K05, 90C25, 90C06, 49M27

1 Introduction and problem setting

Random block-coordinate descent algorithms are nowadays among the methods of choice for solving large scale optimization problems [28, 34, 41]. Indeed, they have low complexity and low memory requirements and, additionally, they are amenable for distributed and parallel implementations [32, 34]. In the last decade a number of works have appeared on the topic which address several aspects, that is: the way the block sampling is performed, the composite structure, the partial separability, and the smoothness/geometrical properties of the objective function, accelerations, and iteration complexity [4, 5, 13, 21, 23, 24, 28, 30, 31, 33, 34, 38].

In this work we consider the following optimization problem

[TABLE]

where $\mathsf{H}$ is the direct sum of $m$ separable real Hilbert spaces $(\mathsf{H}_{i})_{1\leq i\leq m}$, that is,

[TABLE]

and the following assumptions hold:

1. \rmH1

${\mathsf{f}}\colon\mathsf{H}\to\mathbb{R}$ is convex and differentiable, 2. \rmH2

for every $i=1,\dots,m$, ${\mathsf{h}}_{i}\colon\mathsf{H}_{i}\to\left]-\infty,+\infty\right]$ is proper, convex, and lower semicontinuous.

The objective of this study is a stochastic algorithm, called parallel random block-coordinate forward-backward algorithm, that depends on a random variable $\varepsilon$ satisfying the following hypothesis

1. \rmH3

$\varepsilon=(\varepsilon_{1},\dots,\varepsilon_{m})$ is a random variable with values in $\{0,1\}^{m}$ such that, for every $i\in\{1,\dots,m\}$, $\mathsf{p}_{i}:=\mathsf{P}(\varepsilon_{i}=1)>0$ and \mathsf{P}\big{(}\varepsilon=(0,\dots,0)\big{)}=0.

Algorithm 1.1**.**

Let $(\varepsilon^{n})_{n\in\mathbb{N}}=(\varepsilon^{n}_{1},\dots,\varepsilon^{n}_{m})_{n\in\mathbb{N}}$ be a sequence of independent copies of $\varepsilon$. Let $(\gamma_{i})_{1\leq i\leq m}\in\mathbb{R}_{++}^{m}$ and $x^{0}=(x^{0}_{1},\dots,x^{0}_{m})\equiv\mathsf{x}^{0}\in\operatorname*{\text{\rm dom}}{{\mathsf{h}}}$ be a constant random variable. Iterate

[TABLE]

For every $n\in\mathbb{N}$, we denote by $\mathfrak{E}_{n}$ the sigma-algebra generated by $\varepsilon^{0},\dots,\varepsilon^{n}$.

In Algorithm 1.1, the role of the random variable $\varepsilon^{n}$ is to select, at iteration $n$, the blocks to update in parallel (those indexed in $\{i\in\{1,\dots,m\}\,\,|\,\varepsilon_{i}^{n}=1\}$). When all block-coordinates are simultaneously updated at each iteration, Algorithm 1.1 reduces to the (deterministic) forward-backward algorithm, which converges only if the stepsizes are appropriately set. More specifically, if $\nabla{\mathsf{f}}$ is $L$-Lipschitz continuous, then convergence is ensured if the stepsizes $\gamma_{i}$ are all equal and strictly less than $2/L$ [6, 7]. This fact is proved by using the so called descent lemma, i.e.,

[TABLE]

Indeed, (1.3) is itself an assumption concerning the smoothness of ${\mathsf{f}}$, since it is well-known to be equivalent to the Lipschitz continuity of the gradient of ${\mathsf{f}}$ [1, Theorem 18.15]. By contrast, when the block-coordinates are updated one by one in a serial manner, it is desirable to allow moving along the block-coordinates with different stepsizes, depending on the Lipschitz constants of the partial gradients of ${\mathsf{f}}$ across the block-coordinates [2, 28]. So, in this case it is more appropriate to assume that a descent lemma holds on each block-coordinate subspace individually, that is,

[TABLE]

where $\mathsf{J}_{i}\mathsf{v}_{i}=(0,\dots,0,\mathsf{v}_{i},0,\dots,0)$ ($\mathsf{v}_{i}$ occurring at the $i$-th position), for some positive constants $L_{i}$’s. In the setting of Algorithm 1.1, multiple block-coordinates may be updated in parallel at each iteration, according to the random sampling $\varepsilon$. Therefore, it is reasonable to assume that there exists $(\nu_{i})_{1\leq i\leq m}\in\mathbb{R}_{++}^{\mathbb{N}}$ so that one of the generalized smoothness conditions below holds

1. \rmS1

$(\forall\,\mathsf{x},\mathsf{v}\in\mathsf{H})\quad\mathsf{E}[{\mathsf{f}}(\mathsf{x}+\varepsilon\odot\mathsf{v})]\leq{\mathsf{f}}(\mathsf{x})+\mathsf{E}[\langle\nabla{\mathsf{f}}(\mathsf{x}),\varepsilon\odot\mathsf{v}\rangle]+\displaystyle\frac{1}{2}\sum_{i=1}^{m}\mathsf{p}_{i}\nu_{i}\lVert\mathsf{v}_{i}\rVert^{2}$, 2. \rmS2

$(\forall\,\mathsf{x},\mathsf{v}\in\mathsf{H})\quad{\mathsf{f}}(\mathsf{x}+\varepsilon\odot\mathsf{v})\leq{\mathsf{f}}(\mathsf{x})+\langle\nabla{\mathsf{f}}(\mathsf{x}),\varepsilon\odot\mathsf{v}\rangle+\displaystyle\frac{1}{2}\sum_{i=1}^{m}\nu_{i}\varepsilon_{i}\lVert\mathsf{v}_{i}\rVert^{2}\quad\mathsf{P}\text{ a.s.}$,

where $\varepsilon\odot\mathsf{v}=(\varepsilon_{i}\mathsf{v}_{i})_{1\leq i\leq m}\in\mathsf{H}$. Conditions 1 and 2 can be interpreted as descent lemmas on random block-coordinate subspaces, depending on the chosen random sampling of the block-coordinates. They reduce to (1.4), with $\nu_{i}=L_{i}$, if the sampling $\varepsilon$ selects only one block at a time almost surely (see Section 3.2). We call $(\nu_{i})_{1\leq i\leq m}$ the smoothness parameters of ${\mathsf{f}}$. Then, similarly to the deterministic case, we will adopt the following stepsize rule

[TABLE]

Another smoothness condition suitable for Algorithm 1.1, which was considered in [23], is

1. \rmS3

$(\forall\,\mathsf{x},\mathsf{v}\in\mathsf{H})\quad{\mathsf{f}}(\mathsf{x}+\mathsf{v})\leq{\mathsf{f}}(\mathsf{x})+\langle\nabla{\mathsf{f}}(\mathsf{x}),\mathsf{v}\rangle+\displaystyle\frac{1}{2}\sum_{i=1}^{m}\nu_{i}\lVert\mathsf{v}_{i}\rVert^{2}$.

Note that, 1 \ \Rightarrow\ 2 \ \Rightarrow\ 1, which in turn implies the Lipschitz continuity of the gradient of ${\mathsf{f}}$ (see Theorem 3.1iv). So, possibly with different values of the $\nu_{i}$’s, the above conditions are all equivalent. The point is that in the parallel setting (where multiple blocks are updated in parallel at each iteration), 1 may be fulfilled with values of $\nu_{i}$ that are much smaller than those related to the other two conditions, ultimately allowing to significantly increase the stepsizes and hence speeding up the convergence. Moreover, 1 makes parallelization particularly effective on problems with a sparse structure and superior to the serial strategy (which updates a single block per iteration). See the discussion after Theorem 4.9. The critical role played by assumption 1 in the analysis of parallel randomized block-coordinate descent methods was pointed out in [31, 34, 35, 38]. There, it was called expected separable overapproximation (ESO) inequality. Condition 2 is new and serves to guarantee that Algorithm 1.1 is almost surely descending (Proposition 4.7), which is a property that is especially relevant when error bound conditions hold (see Section 4.3). Note that in [34] the issue of monotonicity of the algorithm was addressed for each sampling separately without any general guidance. Finally, we stress that, except for [4, 5] (which study the convergence of the iterates only), in all previous works the stepsizes $\gamma_{i}$’s are set equal to $1/\nu_{i}$. This is an unnecessary limitation that we remove, so to match the standard stepsize rule of the forward-backward algorithm [6, 7].

Remark 1.2**.**

For every $i=1,\dots,m$, the canonical embedding of $\mathsf{H}_{i}$ into $\mathsf{H}$ is the operator $\mathsf{J}_{i}\colon\mathsf{H}_{i}\to\mathsf{H}$, $\mathsf{x}\mapsto(0,\dots,0,\mathsf{x},0,\dots,0)$, where $\mathsf{x}$ occurs in the $i$-th position. Then Algorithm 1.1 can be written as

[TABLE]

1.1 Main contributions and comparison to previous work

In the following we summarize the main contributions of this paper, where, for the sake of brevity, we set ${\mathsf{F}}={\mathsf{f}}+{{\mathsf{h}}}$. We assume that 1–1 are satisfied and that 1 is met. Then, the following hold.

• •

Algorithm 1.1 is descending in expectation and $\mathsf{E}[{\mathsf{F}}(x^{n})]-\inf{\mathsf{F}}\to 0$, even if the infimum is not attained. If $\operatorname*{\text{\rm argmin}}{\mathsf{F}}\neq\varnothing$, then $\mathsf{E}[{\mathsf{F}}(x^{n})]-\inf{\mathsf{F}}=o(1/n)$. In addition, a nonasymptotic bound for $\mathsf{E}[{\mathsf{F}}(x^{n})]-\inf{\mathsf{F}}$ of order $O(1/n)$ holds. Finally, there exists a random variable $x_{*}$ with values in $\operatorname*{\text{\rm argmin}}{\mathsf{F}}$ such that $x^{n}\rightharpoonup x_{*}$ $\mathsf{P}$-a.s. See Theorem 4.9.

• •

If ${\mathsf{F}}$ is strongly convex or satisfies an error bound condition of Luo-Tseng type (see condition 1), then the iterates as well as the corresponding function values generated by Algorithm 1.1, converge linearly in expectation. See Theorem 4.10, Theorem 4.16, and Theorem 4.19.

Our results advance the state-of-the-art in the study of random block-coordinate descent methods under several aspects. We comment on this below.

1. While convergence of the function values has been intensively studied in the related literature (see e.g., [16, 21, 23, 28, 29, 34, 35, 38]), surprisingly, in a convex setting, convergence of the iterates has been investigated only recently in [4], but with stepsizes set according to the global Lipschitz constant of $\nabla{\mathsf{f}}$. See also [12] which addresses the convergence of the iterates in the framework of primal-dual algorithms with a serial and uniform block sampling. We improve the existing results, since we show convergence of the iterates for Algorithm 1.1 in an infinite dimensional setting even when the stepsizes are chosen according to the condition 1, which can incorporate the block Lipschitz constants of the gradient of ${\mathsf{f}}$ and is at the basis of the effectiveness of the parallel block-coordinatewise approach.
2. The worst case asymptotic rate $o(1/n)$ for the mean of the function values is new in the setting of stochastic algorithms.
3. Our analysis spotlights an abstract convergence principle for stochastic descent algorithms (Theorem 4.1) which is essentially a special form of the stochastic quasi-Fejér monotonicity property, involving also the values of the objective functions. This principle, previously investigated in a deterministic setting in [36], allows to prove in a unified way both the almost sure convergence of the iterates and rates of convergence for the mean of the function values.
4. As a by-product of the above analysis we single out an inequality (Proposition 4.4) which is pivotal for studying the convergence under error bound conditions, improving the results and simplifying the analysis in [23].
5. We allow for parallel and arbitrary sampling of the blocks in a composite setting. The benefit of such sampling in terms of convergence rate have been first investigated in [35] for a strongly convex and smooth objective function. In [30] a composite objective optimization problem was analyzed but for a slightly different algorithm. The rest of the studies deal either with parallel uniform sampling of the blocks [34], or with the case where a single block is updated at each iteration [21, 28].
6. We also allow for stepsizes larger than those considered in literature [16, 21, 23, 28, 29, 34, 35, 38], since we can let the stepsizes go beyond $1/\nu_{i}$ and be arbitrarily close to $2/\nu_{i}$, matching the standard rule for the forward-backward algorithm. This provides additional flexibility to the algorithm. Indeed, in the strongly convex case we show that the optimal stepsizes are strictly larger than $1/\nu_{i}$.

The rest of the paper is organized as follows. In Section 2 we give notation and basic facts. Section 3 shows how to determine the smoothness parameters $\nu_{i}$ when ${\mathsf{f}}$ features a partially separable structure. In Section 4 we carry out the convergence analysis and give the related theorems. Finally, Section 5 shows three applications and Section 6 provides some numerical experiments.

2 Notation and background

Notation.

We define $\mathbb{R}_{+}=\left[0,+\infty\right[$, $\mathbb{R}_{++}=\left]0,+\infty\right[$, for every integer $s\geq 1$, $[s]=\{1,\dots,s\}$, and for every $a\in\mathbb{R}^{s}$, $\mathrm{spt}(a)=\{i\in[s]\,|\,a_{i}\neq 0\}$. Scalar products and norms in Hilbert spaces are denoted by $\langle\cdot,\cdot\rangle$ and $\lVert\cdot\rVert$ respectively. If $\mathsf{U}\colon\mathsf{H}\to\mathsf{G}$ is a bounded linear operator between real Hilbert spaces, $\mathsf{U}^{\top}\colon\mathsf{G}\to\mathsf{H}$ is its transpose operator, that is, the one satisfying $\langle\mathsf{U}\mathsf{x},\mathsf{y}\rangle=\langle\mathsf{x},\mathsf{U}^{\top}\mathsf{y}\rangle$, for every $(\mathsf{x},\mathsf{y})\in\mathsf{H}\times\mathsf{G}$. Let $(\mathsf{H}_{i})_{1\leq i\leq m}$ be $m$ separable real Hilbert spaces and let $\mathsf{H}=\bigoplus_{i=1}^{m}\mathsf{H}_{i}$ be their direct sum. For every $\mathsf{v}\in\mathsf{H}$ and $\epsilon\in\{0,1\}^{m}$ we set $\epsilon\odot\mathsf{v}=(\epsilon_{i}\mathsf{v}_{i})_{1\leq i\leq m}\in\mathsf{H}$. We will consider random variables with underlying probability space $(\Omega,\mathfrak{A},\mathsf{P})$ taking values in $\mathsf{H}_{i}$ or $\mathsf{H}$. We use the default font for random variables and sans serif font for their realizations. The expected value operator is denoted by $\mathsf{E}$. A copy of a random variable is random variable having the same distribution of the given one. Let $(w_{i})_{1\leq i\leq m}\in\mathbb{R}^{m}_{++}$. The direct sum operator ${\mathsf{W}}=\bigoplus_{i=1}^{m}w_{i}\mathsf{Id}_{i}$, where $\mathsf{Id}_{i}$ is the identity operator on $\mathsf{H}_{i}$, is the positive bounded linear operator on $\mathsf{H}$ acting as $\mathsf{x}=(\mathsf{x}_{i})_{1\leq i\leq m}\mapsto(w_{i}\mathsf{x}_{i})_{1\leq i\leq m}$. ${\mathsf{W}}$ defines an equivalent inner product on $\mathsf{H}$

[TABLE]

which gives the norm $\lVert\mathsf{x}\rVert_{{\mathsf{W}}}^{2}=\sum_{i=1}^{m}w_{i}\lVert\mathsf{x}_{i}\rVert^{2}$. If $\mathsf{S}\subset\mathsf{H}$ and $\mathsf{x}\in\mathsf{H}$, we set $\mathrm{dist}_{{\mathsf{W}}}(\mathsf{x},\mathsf{S})=\inf_{\bm{\mathsf{z}}\in\mathsf{S}}\lVert\mathsf{x}-\bm{\mathsf{z}}\rVert_{{\mathsf{W}}}$. Let ${\varphi}\colon\mathsf{H}\to\left]-\infty,+\infty\right]$ be proper, convex, and lower semicontinuous. The domain of ${\varphi}$ is $\operatorname*{\text{\rm dom}}{\varphi}=\{\mathsf{x}\in\mathsf{H}\,|\,{\varphi}(\mathsf{x})<+\infty\}$ and the set of minimizers of ${\varphi}$ is $\operatorname*{\text{\rm argmin}}{\varphi}=\{\mathsf{x}\in\mathsf{H}\,|\,{\varphi}(\mathsf{x})=\inf{\varphi}\}$. The subdifferential of ${\varphi}$ in the metric $\langle\cdot,\cdot\rangle_{{\mathsf{W}}}$ is the multivalued operator

[TABLE]

In case ${\mathsf{W}}=\mathsf{Id}$, it is simply denoted by $\partial{\varphi}$. Clearly $\partial^{{\mathsf{W}}}{\varphi}={\mathsf{W}}^{-1}\partial{\varphi}$. If the function ${\varphi}\colon\mathsf{H}\to\mathbb{R}$ is differentiable, then, for every $\mathsf{x}\in\mathsf{H}$, $\partial^{{\mathsf{W}}}{\varphi}(\mathsf{x})=\{\nabla^{{\mathsf{W}}}{\varphi}(\mathsf{x})\}$ and for all $\mathsf{v}\in\mathsf{H}$, $\langle\nabla^{{\mathsf{W}}}{\varphi}(\mathsf{x}),\mathsf{v}\rangle_{{\mathsf{W}}}=\langle\nabla\varphi(\mathsf{x}),\mathsf{v}\rangle$. The proximity operator of ${\varphi}$ in the metric $\langle\cdot,\cdot\rangle_{{\mathsf{W}}}$ is defined as

[TABLE]

Referring to the functions in (1.1), we denote by $\mu_{\mathsf{\Gamma}^{-1}}$ and $\sigma_{\mathsf{\Gamma}^{-1}}$ the moduli of strong convexity of ${\mathsf{f}}$ and ${{\mathsf{h}}}$ respectively, in the norm $\lVert\cdot\rVert_{{\mathsf{\Gamma}}^{-1}}$, where ${\mathsf{\Gamma}}=\bigoplus_{i=1}^{m}\gamma_{i}\mathsf{Id}_{i}$ and the $\gamma_{i}$’s are the stepsizes occurring in Algorithm 1.1. This means that $\mu_{\mathsf{\Gamma}^{-1}},\sigma_{\mathsf{\Gamma}^{-1}}\in\mathbb{R}_{+}$ and that, for every $\mathsf{x},\mathsf{y}\in\mathsf{H}$,

[TABLE]

Note that, since ${{\mathsf{h}}}$ is separable, by taking $\mathsf{y}=\mathsf{x}+\mathsf{J}_{i}(\mathsf{y}_{i}-\mathsf{x}_{i})$ in (2.2), we have

[TABLE]

Remark 2.1**.**

If 1 is satisfied, the $\gamma_{i}$’s are chosen as in (1.5), and $\delta=\max_{1\leq i\leq m}\gamma_{i}\nu_{i}$ (according to the convergence theorems), then we have

[TABLE]

Indeed, let $\mathsf{x}\in\mathsf{H}$, $i\in[m]$ and $\mathsf{v}_{i}\in\mathsf{H}_{i}$, $\mathsf{v}_{i}\neq 0$. It follows from (2.1) with $\mathsf{y}=\mathsf{x}+\varepsilon\odot\mathsf{J}_{i}\mathsf{v}_{i}$ (where $\mathsf{J}_{i}$ is defined in Remark 1.2) and 1 that

[TABLE]

Thus, (2.4) follows.

Fact 2.2** ([10, Example 5.1.5]).**

Let $\zeta_{1}$ and $\zeta_{2}$ be independent random variables with values in the measurable spaces $\mathcal{Z}_{1}$ and $\mathcal{Z}_{2}$ respectively. Let $\varphi\colon\mathcal{Z}_{1}\times\mathcal{Z}_{2}\to\mathbb{R}$ be measurable and suppose that $\mathsf{E}[\lvert\varphi(\zeta_{1},\zeta_{2})\rvert]<+\infty$. Then $\mathsf{E}[\varphi(\zeta_{1},\zeta_{2})\,|\,\zeta_{1}]=\psi(\zeta_{1})$, where for all $z_{1}\in\mathcal{Z}_{1}$, $\psi(z_{1})=\mathsf{E}[\varphi(z_{1},\zeta_{2})]$.

Fact 2.3**.**

Let $\varepsilon$ be a random variable with values in $\{0,1\}^{m}$ and, for all $i\in[m]$, $\mathsf{p}_{i}=\mathsf{P}(\varepsilon_{i}=1)$. Then $\mathsf{E}[\varepsilon_{i}]=\mathsf{p}_{i}$ and, for every $\mathsf{v}=(\mathsf{v}_{i})_{1\leq i\leq m}\in\mathbb{R}^{m}$, $\mathsf{E}[\langle\varepsilon,\mathsf{v}\rangle]=\sum_{i=1}^{m}\mathsf{p}_{i}\mathsf{v}_{i}$.

Fact 2.4** ([17]).**

Let $(a_{n})_{n\in\mathbb{N}}$ be a decreasing sequence in $\mathbb{R}_{+}$. If $\sum_{n=0}^{+\infty}a_{n}<+\infty$, then, for every $n\in\mathbb{N}$, $a_{n}\leq(1/(n+1))\sum_{n=0}^{+\infty}a_{n}$ and a_{n}=o\big{(}1/(n+1)\big{)}.

3 Determining the smoothness parameters

In this section we provide few scenarios for which the relaxed smoothness conditions 1 and 2 can be fully exploited, attaining tight values for the $\nu_{i}$’s. This ultimately allows to take larger stepsizes and improves rates of convergence. In [31, 38] an extensive analysis of cases in which 1 is satisfied is presented.

3.1 General estimates.

We consider the following setting.

1. \rmH4

The function ${\mathsf{f}}\colon\mathsf{H}\to\mathbb{R}$ is such that

[TABLE]

where, for every $k=1,\dots,p$, ${\mathsf{g}}_{k}\colon\mathsf{G}_{k}\to\mathbb{R}$ is a convex differentiable function defined on a real Hilbert space $\mathsf{G}_{k}$ and, for every $i\in[m]$, $\mathsf{U}_{k,i}\colon\mathsf{H}_{i}\to\mathsf{G}_{k}$ is a bounded linear operator. Moreover, $\bigcup_{k=1}^{p}I_{k}\neq\varnothing$, where, for all $k=1,\dots,p$, I_{k}=\big{\{}i\in[m]\,|\,\mathsf{U}_{k,i}\neq 0\big{\}}, and $\eta=\max_{1\leq k\leq p}\mathrm{card}(I_{k})$.

We will also consider one of the following conditions.

1. \rmL1

For every $i=1,\dots,m$ there exists $L_{i}>0$ such that, for every $\mathsf{x}\in\mathsf{H}$, the function $\nabla_{i}{\mathsf{f}}(\mathsf{x}_{1},\dots,\mathsf{x}_{i-1},\cdot,\mathsf{x}_{i+1},\dots,\mathsf{x}_{m})\colon\mathsf{H}_{i}\to\mathsf{H}_{i}$ is $L_{i}$-Lipschitz continuous. 2. \rmL2

For every $k=1,\dots,p$, $\nabla{\mathsf{g}}_{k}\colon\mathsf{G}_{k}\to\mathsf{G}_{k}$ is $L^{(k)}$-Lipschitz continuous and for every $i,j\in[m]$, $i\neq j$, the ranges of $\mathsf{U}_{k,i}$ and $\mathsf{U}_{k,j}$ are orthogonal.

Assumption 1 concerns the partial separability of the function ${\mathsf{f}}$. Depending on the number of the nonzero operators $\mathsf{U}_{k,i}$, ${\mathsf{g}}_{k}$ might depend only on few block-variables $\mathsf{x}_{i}$’s: if $\eta=1$, ${\mathsf{f}}$ is fully separable, whereas if $\eta=m$, ${\mathsf{f}}$ is not separable. Note that 1 is equivalent to (1.4) and, since ${\mathsf{f}}$ is convex, implies the global Lipschitz continuity of the gradient of ${\mathsf{f}}$ (Corollary A.2). So either 1 or 2 implies the global Lipschitz smoothness of ${\mathsf{f}}$. However, considering the constants $L_{i}$’s or $L^{(k)}$’s leads in general to a finer analysis of the smoothness properties of ${\mathsf{f}}$, eventually determining parameters $\nu_{i}$ that are smaller than the global Lipschitz constant of $\nabla{\mathsf{f}}$. Instances of problem (1.1) where ${\mathsf{f}}$ has the structure shown in 1, occur very often in applications. In particular, a prominent example is that of the Lasso problem which will be discussed in Section 5.1. The following theorem, which is proved in Appendix B, relates the smoothness parameters $(\nu_{i})_{1\leq i\leq m}$ to the block Lipschitz constants of the partial gradients of ${\mathsf{f}}$ and to the Lipschitz constants of the gradients of its components ${\mathsf{g}}_{k}$’s in (3.1), as well as to the distribution of the random variable $\varepsilon$.

Theorem 3.1**.**

Assume 1 and 1 and let $(\nu_{i})_{1\leq i\leq m}\in\mathbb{R}_{++}^{\mathbb{N}}$. Then the following hold.

1. (i)

1\ \Rightarrow\* 1 provided that*

[TABLE] 2. (ii)

1\ \Rightarrow\* 2 provided that*

[TABLE] 3. (iii)

2\ \Rightarrow\* 1 provided that*

[TABLE] 4. (iv)

1\ \Rightarrow\1* with, for all $i\in[m]$, $L_{i}=\nu_{i}$. In particular, 1 implies that ${\mathsf{f}}$ is Lipschitz smooth.*

Remark 3.2**.**

1. (i)

Suppose that in 1, for all $k\in[p]$, $\mathsf{G}_{k}=\mathsf{H}$, ${\mathsf{g}}_{k}$ is $L^{(k)}$-Lipschitz smooth, and, for all $i\in I_{k}$, $\mathsf{U}_{k,i}=\mathsf{J}_{i}$, the canonical embedding of $\mathsf{H}_{i}$ into $\mathsf{H}$ (see Remark 1.2). Then, 2 holds and, for every $i\in[m]$, $\tilde{L}_{i}=\sum_{k\,|i\in I_{k}}L^{(k)}$. Hence, in view of Theorem 3.1iii, 1 is met with $\nu_{i}=\tilde{L}_{i}$. This setting was studied in [23]. 2. (ii)

If $\nabla{\mathsf{f}}$ is $L$-Lipschitz continuous, then 1 is satisfied with, for every $i\in[m]$, $\nu_{i}=L$. Therefore, we cover the analysis of the random block-coordinate forward-backward algorithm given in [4, 5] which set the stepsizes as $\gamma_{i}<2/L$. 3. (iii)

Let, for every $k\in[p]$, {\mathsf{f}}_{k}(\mathsf{x})={\mathsf{g}}_{k}\big{(}\sum_{i=1}^{m}\mathsf{U}_{k,i}\mathsf{x}_{i}\big{)}. If, for every $k\in[p]$, 1 (resp. 2) holds for ${\mathsf{f}}_{k}$ with $(\nu_{i}^{(k)})_{1\leq i\leq m}$ , then 1 (resp. 2) holds for ${\mathsf{f}}$ with $\nu_{i}=\sum_{k=1}^{p}\nu_{i}^{(k)}$. 4. (iv)

Using similar ideas as in the proof of [34, Theorem 12] we show in Appendix B that item i in Theorem 3.1 remains true with

[TABLE]

Remark 3.3**.**

Referring to Theorem 3.1, for all $i\in[m]$, we have $1\leq\beta_{1,i}\leq\beta_{2}\leq\min\{\eta,\tau_{\max}\}$, where

[TABLE]

is the maximum number of blocks processed in parallel. Indeed, since $\mathsf{P}(\varepsilon\equiv 0)=0$ we have $\mathsf{P}(\max_{1\leq i\leq m}\varepsilon_{i}\geq 1)=1$. Moreover, since \max_{1\leq k\leq p}\big{(}\sum_{i\in I_{k}}\varepsilon_{i}\big{)}\geq\max_{1\leq i\leq m}\varepsilon_{i}, we have $1\leq\beta_{1,i}$. The inequality $\beta_{1,i}\leq\beta_{2}$ is immediate, while the last one derives from the following

[TABLE]

3.2 The smoothness parameters for some special block samplings.

Here we show how to compute (or estimate) the constants $(\beta_{1,i})_{1\leq i\leq m}$ and $\beta_{2}$ in Theorem 3.1 and Remark 3.2iv, and the related $(\nu_{i})_{1\leq i\leq m}$, in some relevant scenarios, when 1 and 1 are satisfied.

Arbitrary parallel sampling.

It follows from Theorem 3.1ii and Remark 3.3 that for an arbitrary (possibly nonuniform) block sampling $\varepsilon$, 2 is satisfied provided that $\nu_{i}=\min\{\eta,\tau_{\max}\}L_{i}$, for every $i\in[m]$. Additionally, if we denote by $L^{(k)}_{i}$ the blockwise Lipschitz constants of the gradient of the function \mathsf{x}\mapsto{\mathsf{g}}_{k}\big{(}\sum_{i=1}^{m}\mathsf{U}_{k,i}\mathsf{x}_{i}\big{)}, then we derive from Remark 3.2iii and the above discussion that 2 holds with111If $i\notin I_{k}$, then $L^{(k)}_{i}=0$. $\nu_{i}=\sum_{k|i\in I_{k}}\min\{\mathrm{card}(I_{k}),\tau_{\max}\}L^{(k)}_{i}$. However, the above estimates are rather conservative and can be improved for special choices of the block sampling as we will show below. We refer to [31, 35] for further results on nonuniform samplings.

Serial sampling or full separability.

Suppose that $\tau_{\max}=1$ or $\eta=1$. Then, Remark 3.3 yields $\beta_{1,i}=\beta_{2}=\min\{\eta,\tau_{\max}\}=1$. Moreover, recalling Theorem 3.1ii-iv, this also shows that 1, 2, and 1 are indeed equivalent with the same smoothness parameters $\nu_{i}=L_{i}$. So, conditions 1 or 2 find their justification only in the parallel case ($\tau_{\max}>1$) and when ${\mathsf{f}}$ is not fully separable ($\eta>1$).

Fully Parallel.

If $\mathsf{P}(\sum_{i=1}^{m}\varepsilon_{i}=m)=1$, then for every $i\in[m]$ $\mathsf{p}_{i}=1$. This yields a fully parallel (deterministic) algorithm. Moreover, since \mathsf{P}\big{(}\varepsilon=(1,\dots,1)\big{)}=1, we have $\beta_{1,i}=\eta=\beta_{2}$ and hence 2 holds with $\nu_{i}=\eta L_{i}$. Actually, also 1 holds with $\nu_{i}=\eta L_{i}$ (see Corollary A.2iv).

Uniform samplings.

Suppose that $m>1$. The sampling is uniform if $\mathsf{p}_{i}=\mathsf{p}_{j}$, with $i\neq j$. In this case if we denote by $\bar{\tau}$ the average number of block updates per iteration, we have $\bar{\tau}=\mathsf{E}[\sum_{i=1}^{m}\varepsilon_{i}]=\sum_{i=1}^{m}\mathsf{p}_{i}$ and hence $\mathsf{p}_{i}=\bar{\tau}/m$, for every $i\in[m]$. In [34] several types of uniform samplings are studied. In the following we single out two of them. The sampling is said to be doubly uniform if any two sets of blocks with the same number of blocks have the same probability to be chosen. In formula, this means that for every $J_{1},J_{2}\subset[m]$ such that $\mathrm{card}(J_{1})=\mathrm{card}(J_{2})$, $\mathsf{P}(\cap_{i\in J_{1}}\{\varepsilon_{i}=1\})=\mathsf{P}(\cap_{i\in J_{2}}\{\varepsilon_{i}=1\})$. For such sampling one directly derives from (3.3) in Remark 3.2iv (see Appendix B) that

[TABLE]

A special type of doubly uniform sampling is the $\tau$-nice sampling in which $\sum_{i=1}^{m}\varepsilon_{i}=\tau$ $\mathsf{P}$-a.s. for some $\tau\in[m]$. In this case (3.4) reduces to

[TABLE]

Now, according to Remark 3.2iv, if we set, for every $i\in[m]$, $\nu_{i}=\beta_{1}L_{i}$, then condition 1 holds. Additionally, if we denote by $L^{(k)}_{i}$ the blockwise Lipschitz constants of the gradient of the function \mathsf{x}\mapsto{\mathsf{g}}_{k}\big{(}\sum_{i=1}^{m}\mathsf{U}_{k,i}\mathsf{x}_{i}\big{)}, then we derive from Remark 3.2iii and (3.5) that 1 is satisfied with $\nu_{i}=\sum_{k|i\in I_{k}}(1+(\tau-1)(\mathrm{card}(I_{k})-1)/(m-1))L^{(k)}_{i}$. This result provides possibly even smaller values for the parameters $(\nu_{i})_{1\leq i\leq m}$ and was given, in the special setting of Remark 3.2i, in [13, 38].

4 Convergence analysis

In the rest of the paper, referring to Algorithm 1.1, we set

[TABLE]

where $\mathsf{Id}_{i}$ is the identity operator on $\mathsf{H}_{i}$, and

[TABLE]

Then, we have

[TABLE]

and, recalling (1.2), that for every $i\in[m]$ such that $\varepsilon^{n}_{i}=1$,

[TABLE]

Note that $x^{n}$ and $\bar{x}^{n+1}$ are functions of the random variables $\varepsilon^{0},\dots,\varepsilon^{n-1}$ only, hence they are both discrete random variables, which are measurable with respect to $\mathfrak{E}_{n-1}$.

4.1 An abstract principle for stochastic convergence

We provide an abstract convergence principle for stochastic descent algorithms in the same spirit of [36, Theorem 3.10]. It simultaneously addresses the convergence of the iterates and that of the function values.

Theorem 4.1**.**

Let $\mathsf{H}$ be a separable real Hilbert space with norm $\lVert\cdot\rVert$. Let ${\mathsf{\Phi}}\colon\mathsf{H}\to\left]-\infty,+\infty\right]$ be a proper, lower semicontinuous, and convex function and set $\mathsf{S}_{*}=\operatorname*{\text{\rm argmin}}{\mathsf{\Phi}}$ and ${\mathsf{\Phi}}_{*}=\inf{\mathsf{\Phi}}$. Let $(x^{n})_{n\in\mathbb{N}}$ be a sequence of $\mathsf{H}$-valued random variables such that $x^{0}\equiv\mathsf{x}^{0}\in\operatorname*{\text{\rm dom}}{\mathsf{\Phi}}$ and, for every $n\in\mathbb{N}$, ${\mathsf{\Phi}}(x^{n})$ is $\mathsf{P}$-summable. Consider the following conditions

1. \rmP1

$(\mathsf{E}[{\mathsf{\Phi}}(x^{n})])_{n\in\mathbb{N}}$* is decreasing.* 2. \rmP2

There exist a sequence $(\mathfrak{X}_{n})_{n\in\mathbb{N}}$ of sub-sigma algebras of $\mathfrak{A}$ such that, $(\forall\,n\in\mathbb{N})$ $\mathfrak{X}_{n}\subset\mathfrak{X}_{n+1}$ and $x^{n}$ is $\mathfrak{X}_{n}$-measurable, a sequence $(\xi_{n})_{n\in\mathbb{N}}$ of $\mathfrak{X}_{n}$-measurable real-valued positive random variables such that $\sum_{n\in\mathbb{N}}\mathsf{E}[\xi_{n}]\leq b<+\infty$, and $a>0$ such that, for every $\mathsf{x}\in\operatorname*{\text{\rm dom}}{\mathsf{\Phi}}$ and $n\in\mathbb{N}$,

[TABLE] 3. \rmP3

There exist $(y^{n})_{n\in\mathbb{N}}$ and $(v^{n})_{n\in\mathbb{N}}$, sequences of $\mathsf{H}$-valued random variables, such that $(\forall\,n\in\mathbb{N})$ $v^{n}\in\partial{\mathsf{\Phi}}(y^{n})$, $y^{n}-x^{n}\rightharpoonup 0$, and $v^{n}\to 0$ $\mathsf{P}$-a.s.

Assume 1 and that (\inf_{n\in\mathbb{N}}\mathsf{E}[{\mathsf{\Phi}}(x^{n})]>-\infty)\Rightarrow\2. Then, the following hold.

1. (i)

$\mathsf{E}[{\mathsf{\Phi}}(x^{n})]\to{\mathsf{\Phi}}_{*}$. 2. (ii)

Suppose that $\mathsf{S}_{*}\neq\varnothing$. Then $\mathsf{E}[{\mathsf{\Phi}}(x^{n})]-{\mathsf{\Phi}}_{*}=o(1/n)$ and,

[TABLE] 3. (iii)

Suppose that 3 holds and $\mathsf{S}_{*}\neq\varnothing$. Then, there exists a random variable $x_{*}$ taking values in $\mathsf{S}_{*}$ such that $x^{n}\rightharpoonup x_{*}$ $\mathsf{P}$-a.s.

Proof.

Taking the expectation in (4.5), we obtain

[TABLE]

i: Since $(\mathsf{E}[{\mathsf{\Phi}}(x^{n})])_{n\in\mathbb{N}}$ is decreasing, $\mathsf{E}[{\mathsf{\Phi}}(x^{n})]\to\inf_{n\in\mathbb{N}}\mathsf{E}[{\mathsf{\Phi}}(x^{n})]\geq{\mathsf{\Phi}}_{*}$. Thus, the statement is true if $\inf_{n\in\mathbb{N}}\mathsf{E}[{\mathsf{\Phi}}(x^{n})]=-\infty$. Suppose that $\inf_{n\in\mathbb{N}}\mathsf{E}[{\mathsf{\Phi}}(x^{n})]>-\infty$ and let $\mathsf{x}\in\operatorname*{\text{\rm dom}}{\mathsf{\Phi}}$. Then, 2 holds and the right hand side of (4.6), being summable, converges to zero. Therefore, ${\mathsf{\Phi}}_{*}\leq\lim_{n\to+\infty}\mathsf{E}[{\mathsf{\Phi}}(x^{n+1})]\leq{\mathsf{\Phi}}(\mathsf{x})$. Since $\mathsf{x}$ is arbitrary in $\operatorname*{\text{\rm dom}}{\mathsf{\Phi}}$, $\mathsf{E}[{\mathsf{\Phi}}(x^{n})]\to{\mathsf{\Phi}}_{*}$.

ii: Let $\mathsf{x}\in\mathsf{S}_{*}$. Then, $\inf_{n\in\mathbb{N}}\mathsf{E}[{\mathsf{\Phi}}(x^{n})]\geq{\mathsf{\Phi}}(\mathsf{x})>-\infty$. Hence 2 holds and (4.6) yields

[TABLE]

Therefore, $\sum_{n\in\mathbb{N}}(\mathsf{E}[{\mathsf{\Phi}}(x^{n+1})]-{\mathsf{\Phi}}_{*})\leq(1/a)\lVert\mathsf{x}^{0}-\mathsf{x}\rVert^{2}+b/a$. Since $(\mathsf{E}[{\mathsf{\Phi}}(x^{n+1})]-{\mathsf{\Phi}}_{*})_{n\in\mathbb{N}}$ is decreasing, the statement follows from Fact 2.4.

iii: Let $\mathsf{x}\in\mathsf{S}_{*}$. Then 2 holds and, since ${\mathsf{\Phi}}(\mathsf{x})\leq{\mathsf{\Phi}}(x^{n+1})$, we derive from (4.5) that,

[TABLE]

Note that $\xi_{n}$ and $\lVert x^{n}-\mathsf{x}\rVert^{2}$ are $\mathfrak{X}_{n}$-measurable. Moreover $\mathsf{E}[\sum_{n\in\mathbb{N}}\xi_{n}]=\sum_{n\in\mathbb{N}}\mathsf{E}[\xi_{n}]<+\infty$ and hence $\sum_{n\in\mathbb{N}}\xi_{n}<+\infty$ $\mathsf{P}$-a.s. Therefore $(x^{n})_{n\in\mathbb{N}}$ is a stochastic quasi-Fejér sequence with respect to $\mathsf{S}_{*}$ [11]. Then, in view of [4, Proposition 2.3(iv)] it is sufficient to prove that the weak limit points of $(x^{n})_{n\in\mathbb{N}}$ are contained in $\mathsf{S}_{*}$ $\mathsf{P}$-a.s. By assumption 3 there exist two sequences of $\mathsf{H}$-valued random variables $(y^{n})_{n\in\mathbb{N}}$ and $(v^{n})_{n\in\mathbb{N}}$ and $\tilde{\Omega}\subset\Omega$, $\mathsf{P}(\tilde{\Omega})=1$ such that, for every $\omega\in\tilde{\Omega}$, $v^{n}(\omega)\in\partial{\mathsf{\Phi}}(y^{n}(\omega))$, $y^{n}(\omega)-x^{n}(\omega)\rightharpoonup 0$, $v^{n}(\omega)\to 0$. Let $\omega\in\tilde{\Omega}$ and let $(x^{n_{k}}(\omega))_{n\in\mathbb{N}}$ be a subsequence of $(x^{n}(\omega))_{n\in\mathbb{N}}$ such that $x^{n_{k}}(\omega)\rightharpoonup\bar{\mathsf{x}}$, for some $\bar{\mathsf{x}}\in\mathsf{H}$. Then,

[TABLE]

Since $\partial{\mathsf{\Phi}}$ is weakly-strongly closed [1], we have $0\in\partial{\mathsf{\Phi}}(\bar{\mathsf{x}})$, so $\bar{\mathsf{x}}\in\mathsf{S}_{*}$. ∎

Remark 4.2**.**

Inequalities similar to (4.5) appear implicitly in the analysis of several deterministic and stochastic algorithms [3, 18, 26], to get rate of convergence for the function values. Moreover, (4.5) is related also to the concept introduced in [20], in a deterministic setting.

4.2 Convergence under convexity and strong convexity assumptions

In this section we address the convergence of Algorithm 1.1 in the convex and strongly convex case. The main results consist in the $o(1/n)$ rate of convergence for the mean of the function values and in the almost sure weak convergence of the iterates. We start by recalling a standard result (see [36, Lemma 3.12(iii)]). Here we give a slightly more general version, including the moduli of strong convexity. The proof is given in Appendix B for reader’s convenience.

Lemma 4.3**.**

Let $\mathsf{H}$ be a real Hilbert space. Let $\varphi\colon\mathsf{H}\to\mathbb{R}$ be differentiable and convex with modulus of strong convexity $\mu_{\varphi}\geq 0$ and $\psi\colon\mathsf{H}\to\left]-\infty,+\infty\right]$ be proper, lower semicontinuous, and convex with modulus of strong convexity $\mu_{\psi}\geq 0$. Let $\mathsf{x}\in\mathsf{H}$ and set $\mathsf{x}^{+}=\text{\sf prox}_{\psi}(\mathsf{x}-\nabla\varphi(\mathsf{x}))$. Then, for every $\mathsf{z}\in\mathsf{H}$,

[TABLE]

Proposition 4.4**.**

Let 1–1 be satisfied. Let $(\nu_{i})_{1\leq i\leq m}\in\mathbb{R}^{m}_{++}$ and suppose that 1 holds. Let $(x^{n})_{n\in\mathbb{N}}$ be generated by Algorithm 1.1 with, for every $i\in[m]$, $\gamma_{i}<2/\nu_{i}$. Set $\delta=\max_{1\leq i\leq m}\gamma_{i}\nu_{i}$ and $\mathsf{p}_{\min}=\min_{1\leq i\leq m}\mathsf{p}_{i}$. Let $\mathsf{\Gamma}^{-1}$ be as in (4.1) and $\mu_{\mathsf{\Gamma}^{-1}}$ and $\sigma_{\mathsf{\Gamma}^{-1}}$ be the moduli of strong convexity of ${\mathsf{f}}$ and ${{\mathsf{h}}}$ respectively, in the norm $\lVert\cdot\rVert_{\mathsf{\Gamma}^{-1}}$. Set ${\mathsf{F}}={\mathsf{f}}+{{\mathsf{h}}}$. Then,

[TABLE]

Proof.

Let $\mathsf{x}\in\operatorname*{\text{\rm dom}}{\mathsf{F}}$ and $n\in\mathbb{N}$. Since for all $\mathsf{v}\in\mathsf{H}$, $\langle\nabla^{{\mathsf{\Gamma}}^{-1}}{\mathsf{f}}(x^{n}),\mathsf{v}\rangle_{{\mathsf{\Gamma}}^{-1}}=\langle\nabla{\mathsf{f}}(x^{n}),\mathsf{v}\rangle$, we derive from Lemma 4.3, written in the norm $\lVert\cdot\rVert_{{\mathsf{\Gamma}}^{-1}}$, and (4.3) that

[TABLE]

Now, we majorize ${{\mathsf{h}}}(x^{n})-{{\mathsf{h}}}(\bar{x}^{n+1})+\langle\nabla{\mathsf{f}}(x^{n}),x^{n}-\bar{x}^{n+1}\rangle$. By Fact 2.2 and Fact 2.3, we have

[TABLE]

Moreover,

[TABLE]

where in the last inequality we used that

[TABLE]

which was obtained from (2.3) with

[TABLE]

Therefore,

[TABLE]

Next, it follows from (4.3), 1, and Fact 2.2 that

[TABLE]

Then, we derive from (4.11) that

[TABLE]

The statement follows from (4.9), considering that

[TABLE]

Proposition 4.5**.**

Let 1–1 be satisfied. Let ${\mathsf{\Gamma}}^{-1}$ and ${\mathsf{W}}$ be as in (4.1) and $(x^{n})_{n\in\mathbb{N}}$ be generated by Algorithm 1.1. Let $n\in\mathbb{N}$ and $x$ be an $\mathsf{H}$-valued random variable which is measurable w.r.t. $\mathfrak{E}_{n-1}$. Then

[TABLE]

and $\mathsf{E}[\lVert x^{n+1}-x^{n}\rVert_{\mathsf{W}}^{2}\,|\,\mathfrak{E}_{n-1}]=\lVert\bar{x}^{n+1}-x^{n}\rVert^{2}_{{\mathsf{\Gamma}}^{-1}}$.

Proof.

If follows from (4.3), Fact 2.2, and Fact 2.3 that

[TABLE]

The second equation follows from (4.12), by choosing $x=x^{n}$. ∎

The following result is a stochastic version of [36, Proposition 3.15].

Proposition 4.6**.**

Let 1–1 be satisfied. Let $(\nu_{i})_{1\leq i\leq m}\in\mathbb{R}^{m}_{++}$ and suppose that 1 holds. Let $(x^{n})_{n\in\mathbb{N}}$ be generated by Algorithm 1.1 with, for every $i\in[m]$, $\gamma_{i}<2/\nu_{i}$. Set $\delta=\max_{1\leq i\leq m}\gamma_{i}\nu_{i}$ and $\mathsf{p}_{\min}=\min_{1\leq i\leq m}\mathsf{p}_{i}$. Let $\mathsf{\Gamma}^{-1}$ and ${\mathsf{W}}$ be as in (4.1) and $\mu_{\mathsf{\Gamma}^{-1}}$ and $\sigma_{\mathsf{\Gamma}^{-1}}$ be the moduli of strong convexity of ${\mathsf{f}}$ and ${{\mathsf{h}}}$ respectively, in the norm $\lVert\cdot\rVert_{\mathsf{\Gamma}^{-1}}$. Set ${\mathsf{F}}={\mathsf{f}}+{{\mathsf{h}}}$. Then, the following hold.

1. (i)

$(\mathsf{E}[{\mathsf{F}}(x^{n})])_{n\in\mathbb{N}}$* is decreasing.* 2. (ii)

Suppose that $\inf_{n\in\mathbb{N}}\mathsf{E}[{\mathsf{F}}(x^{n})]>0$. Then,

[TABLE] 3. (iii)

For every $n\in\mathbb{N}$ and every $\mathsf{x}\in\operatorname*{\text{\rm dom}}{\mathsf{F}}$

[TABLE]

Proof.

Let $n\in\mathbb{N}$ and $\mathsf{x}\in\operatorname*{\text{\rm dom}}{\mathsf{F}}$. Since

[TABLE]

we derive from (4.8), multiplied by $2$, that

[TABLE]

Then for an $\mathsf{H}$-valued $\mathfrak{E}_{n-1}$-measurable random variable $x$, Proposition 4.5 yields

[TABLE]

Taking $x=x^{n}$ in (4.14), we have

[TABLE]

which plugged into (4.14), with $x\equiv\mathsf{x}\in\operatorname*{\text{\rm dom}}{\mathsf{F}}$, gives iii. Moreover, taking the expectation in (4.15), we obtain

[TABLE]

which gives i. Finally, set for all $n\in\mathbb{N}$, \xi_{n}=\mathsf{E}\big{[}{\mathsf{F}}(x^{n})-{\mathsf{F}}(x^{n+1})\big{|}\mathfrak{E}_{n-1}\big{]}\geq 0. Then

[TABLE]

This shows that if $\inf_{n\in\mathbb{N}}\mathsf{E}[{\mathsf{F}}(x^{n})]>0$, then $\sum_{n=0}^{+\infty}\xi_{n}$ is $\mathsf{P}$-integrable and hence it is $\mathsf{P}$-a.s. finite. Then ii follows from (4.15) and Proposition 4.5. ∎

Proposition 4.7**.**

Under the same assumptions of Proposition 4.6, suppose that condition 1 is replaced by condition 2. Then

[TABLE]

Proof.

We derive from 2 (since $\varepsilon^{n}$ has the same distribution of $\varepsilon$) and (4.3) that

[TABLE]

Therefore, summing (4.10), from $i=1$ to $m$, we have

[TABLE]

Hence $(1/2)\sum_{i=1}^{m}(2+\sigma_{\mathsf{\Gamma}^{-1}}-\gamma_{i}\nu_{i})\gamma_{i}^{-1}\varepsilon_{i}^{n}\lVert\Delta_{i}^{n}\rVert^{2}\leq{\mathsf{F}}(x^{n})-{\mathsf{F}}(x^{n+1})$ $\mathsf{P}$-a.s. ∎

Proposition 4.8**.**

Under the assumptions of Proposition 4.6, suppose in addition that ${\mathsf{F}}$ is bounded from below. Then, there exist $(y^{n})_{n\in\mathbb{N}}$ and $(v^{n})_{n\in\mathbb{N}}$, sequences of $\mathsf{H}$-valued random variables, such that the following hold.

1. (i)

$v^{n}\in\partial{\mathsf{F}}(y^{n})$* $\mathsf{P}$-a.s.* 2. (ii)

$y^{n}-x^{n}\to 0$* and $v^{n}\to 0$ $\mathsf{P}$-a.s.*

Proof.

It follows from (4.2) that, $(x_{i}^{n}(\omega)-\bar{x}_{i}^{n+1}(\omega))/\gamma_{i}-\nabla_{i}{\mathsf{f}}(x^{n}(\omega))\in\partial{\mathsf{h}}_{i}(\bar{x}_{i}^{n+1}(\omega))$, for all $i\in[m]$ and $\omega\in\Omega$. Hence

[TABLE]

Set $y^{n}=\bar{x}^{n+1}$ and let $v^{n}\colon\Omega\to\mathsf{H}$ be such that, for every $\omega\in\Omega$,

[TABLE]

Clearly $v^{n}$ is measurable and hence it is a random variable. Moreover, for every $\omega\in\Omega$,

[TABLE]

Now, since ${\mathsf{F}}$ is bounded from below, Proposition 4.6ii yields that $(\lVert y^{n}-x^{n}\rVert_{\mathsf{\Gamma}^{-1}}^{2})_{n\in\mathbb{N}}$ is summable $\mathsf{P}$-a.s. and hence $y^{n}-x^{n}\to 0$ $\mathsf{P}$-a.s. The statement follows from the fact that $\nabla{\mathsf{f}}$ is Lipschitz continuous (see Theorem 3.1iv). ∎

Now we are ready to state one of the main convergence results of this paper. From one hand, it extends to the stochastic setting a well-known convergence rate of the (deterministic) forward-backward algorithm [7, 15, 36]. On the other hand, it proves the almost sure weak convergence of the iterates of Algorithm 1.1 in the convex case. We stress that none of the works [21, 23, 31, 33, 34, 35, 38] addresses this latter aspect. To the best of our knowledge, [4] is the only work that proves almost sure weak convergence of the iterates. However, in [4, Corollary 5.11] the stepsize is set according to the (global) Lipschitz constant of $\nabla{\mathsf{f}}$ which, in general, leads to smaller stepsizes and worse upper bounds on convergence rates. See the subsequent discussion.

Theorem 4.9**.**

Let 1–1 be satisfied. Let $(\nu_{i})_{1\leq i\leq m}\in\mathbb{R}^{m}_{++}$ and suppose that 1 holds. Let $(x^{n})_{n\in\mathbb{N}}$ be generated by Algorithm 1.1 with, for every $i\in[m]$, $\gamma_{i}<2/\nu_{i}$. Set $\delta=\max_{1\leq i\leq m}\gamma_{i}\nu_{i}$ and $\mathsf{p}_{\min}=\min_{1\leq i\leq m}\mathsf{p}_{i}$. Let ${\mathsf{W}}$ be as in (4.1) and set ${\mathsf{F}}={\mathsf{f}}+{{\mathsf{h}}}$, ${\mathsf{F}}_{*}=\inf{\mathsf{F}}$, and $\mathsf{S}_{*}=\operatorname*{\text{\rm argmin}}{\mathsf{F}}\subset\mathsf{H}$. Then, the following hold.

1. (i)

$\mathsf{E}[{\mathsf{F}}(x^{n})]\to{\mathsf{F}}_{*}$. 2. (ii)

Suppose that $\mathsf{S}_{*}\neq\varnothing$. Then $\mathsf{E}[{\mathsf{F}}(x^{n})]-{\mathsf{F}}_{*}=o(1/n)$ and, for every integer $n\geq 1$,

[TABLE]

Moreover, there exists a random variable $x_{*}$ taking values in $\mathsf{S}_{*}$ such that $x^{n}\rightharpoonup x_{*}$ $\mathsf{P}$-a.s.

Proof.

Proposition 4.6iii with $\mu_{\mathsf{\Gamma}^{-1}}=\sigma_{\mathsf{\Gamma}^{-1}}=0$ gives, for all $\mathsf{x}\in\operatorname*{\text{\rm dom}}{\mathsf{F}}$ and $n\in\mathbb{N}$,

[TABLE]

where

[TABLE]

Note that the random variables $x^{n}$’s are discrete with finite range and $(\mathsf{E}[{\mathsf{F}}(x^{n})])_{n\in\mathbb{N}}$ is decreasing. Moreover, $\sum_{n\in\mathbb{N}}\mathsf{E}[\xi_{n}]\leq b_{1}({\mathsf{F}}(\mathsf{x}^{0})-\inf_{n\in\mathbb{N}}\mathsf{E}[{\mathsf{F}}(x^{n})])$. Therefore, the statement follows from Theorem 4.1 and Proposition 4.8. ∎

Discussion.

In the following we examine some crucial aspects related to Algorithm 1.1. We suppose that 1 and 1 hold and that for every $i\in[m]$, $\gamma_{i}=\delta/\nu_{i}$ with $\delta\in\left]0,2\right[$.

The benefit of a parallel block update.

Here we discuss the advantage of updating multiple blocks in parallel instead of just a single block. We consider the setting of a $\tau$-nice uniform block sampling, which was described in Section 3.2. In this case, for ever $i,j\in[m]$, with $i\neq j$, $\mathsf{p}_{i}=\mathsf{p}_{j}$ and, since $\tau=\sum_{i=1}^{m}\mathsf{E}[\varepsilon_{i}]=\sum_{i=1}^{m}\mathsf{p}_{i}$, we have, for every $i\in[m]$, $\mathsf{p}_{i}=\mathsf{p}:=\tau/m$. Moreover, we can set for every $i\in[m]$, $\nu_{i}=\beta_{1}L_{i}$ with $\beta_{1}$ defined as in (3.5). In order to compare different choices of $\tau$, we normalize the iterations so to match the same computational cost per iteration of the standard (full parallel) forward-backward algorithm (FB). It follows from (4.18) that after $n_{\tau}=\lceil m\bar{n}/\tau\rceil$ iterations of Algorithm 1.1, which have the same total computational cost of $\bar{n}$ iterations of FB, we have

[TABLE]

where $\mathsf{\Lambda}=\bigoplus_{i=1}^{m}L_{i}\mathsf{Id}_{i}$. Now, since, $\beta_{1}=1+(\tau-1)(\eta-1)/(m-1)$, we see that if $\eta\ll m$ and $\tau\ll m$, then $\beta_{1}$ is close to 1 (and $\mathsf{p}$ is close to zero), so that $\beta_{1}$ nearly does not depend on $\tau$, as long as $\tau$ remains sufficiently small. For instance, in Section 6 we consider the setting where $m=10^{5}$ and $\eta=148$. In such case, if we let $\tau=1,5,10,50$, the corresponding $\beta_{1}$’s and $\mathsf{p}$’s are essentially the same so that the right hand side of (4.19) does not change much. Therefore, the above options for $\tau$ require the same total amount of computations (i.e., $m\bar{n}$ block-coordinate updates) and lead essentially to the same improvement in the objective function. However, a parallel implementation, say with $\tau=50$ on a CPU with $50$ cores, will be $50$ times faster than a serial implementation ($\tau=1$) which uses only one core per iteration. In summary, in the large scale ($m$ large) and sparse ($\eta\ll m$) setting, the parallel strategy ($\tau>1$ and $\tau$ equal to the number of CPU cores) is definitely advantageous provided that $\tau$ is sufficiently small compared to $m$.

Comparison with [4].

The almost sure weak convergence of the iterates of Algorithm 1.1 is also obtained in [4], but with stepsizes set according to the global Lipschitz constant of the gradient of ${\mathsf{f}}$. Let $L$ be the Lipschitz constant of $\nabla{\mathsf{f}}$ and note that ${\mathsf{f}}$ is also Lipschitz smooth in the norm $\lVert\cdot\rVert_{\mathsf{\Lambda}}$, defined by the operator $\mathsf{\Lambda}=\bigoplus_{i=1}^{m}L_{i}\mathsf{Id}_{i}$, with constant $\eta$ (see Corollary A.2iv). Therefore, the results in [4] can be applied in the original norm $\lVert\cdot\rVert$ or in the norm $\lVert\cdot\rVert_{\mathsf{\Lambda}}$. In this respect we note that since

[TABLE]

the implementation in the norm $\lVert\cdot\rVert_{\mathsf{\Lambda}}$ is nothing but Algorithm 1.1 with stepsizes $\gamma_{i}=\delta/(\eta L_{i})$. In both cases Corollary 5.11 in [4], applied in the corresponding norms, proves weak convergence of the iterates for Algorithm 1.1 with stepsizes $\gamma_{i}\equiv\delta/L$ and $\gamma_{i}=\delta/(\eta L_{i})$ respectively. However, Theorem 4.9, together with Theorem 3.1, allows to set the stepsizes as $\gamma_{i}=\delta/(\beta_{1,i}L_{i})$. Since $\beta_{1,i}L_{i}$ may be much smaller than $L$ and, in view of Remark 3.3, it is always smaller than $\eta L_{i}$, Theorem 4.9 provides a significant improvement over [4] in terms of flexibility in the stepsizes.

The advantage over the standard FB.

We consider the forward-backward algorithm (FB) in the original norm of $\mathsf{H}$ and in the norm $\lVert\cdot\rVert_{\mathsf{\Lambda}}$. All the remarks about the stepsizes discussed in the previous paragraph apply also here. Moreover, in the case $\delta=1$, standard convergence rate for FB (see e.g., [7]) yields that after $\bar{n}$ iterations, we have

[TABLE]

depending on which of the two above implementations of FB we consider. In order to appropriately compare the rates (4.20) with that of Algorithm 1.1 given in Theorem 4.9 in the following we set $\delta=1$ and analyze two choices of the block sampling.

1. (i)

Assume that we perform a $\tau$-nice block sampling. Then we saw that the (normalized) convergence rate of Algorithm 1.1 is (4.19). We first note that (4.19) reduces to the second inequality in (4.20) when $\delta=1$ and $\tau=m$. Comparing the bounds in (4.20) with (4.19) (with $\delta=1$) we see that, if we assume that the terms $\mathrm{dist}_{\mathsf{\Lambda}}^{2}(\mathsf{x}^{0},\mathsf{S}_{*})$ and ${\mathsf{F}}(\mathsf{x}^{0})-{\mathsf{F}}_{*}$ are about of the same magnitude, then Algorithm 1.1 features always a better rate than FB if implemented in the norm $\lVert\cdot\rVert_{\mathsf{\Lambda}}$ (since $\beta_{1}\leq\eta$), whereas if FB is implemented in the original norm of $\mathsf{H}$, Algorithm 1.1 is still a better choice provided that $\beta_{1}\max_{1\leq i\leq m}L_{i}\leq L$. 2. (ii)

Suppose that the block sampling performs on average $\tau$ updates per iteration and that, for every $i\in[m]$, $\mathsf{p}_{i}$ is proportional to the Lipschitz constant $L_{i}$, that is, $\mathsf{p}_{i}=\tau L_{i}/(\sum_{j=1}^{m}L_{j})$ (provided that $\tau\leq(\sum_{j=1}^{m}L_{j})/\max_{1\leq j\leq m}L_{j}$). In this case, as stated at the beginning of Section 3.2, we can let $\nu_{i}=\beta_{2}L_{i}$ and $\gamma_{i}=1/\nu_{i}$. Then, (4.18) becomes for $\delta=1$ and $n=\lceil m\bar{n}/\tau\rceil$,

[TABLE]

where $\bar{L}=\sum_{i=1}^{m}L_{i}/m$ and $L_{\min}=\min_{1\leq i\leq m}L_{i}$. Here we see that Algorithm 1.1 can be superior to FB if $(\beta_{2}+2\bar{L}/L_{\min})\bar{L}\leq L$, under the assumption that $\bar{L}\mathrm{dist}^{2}(\mathsf{x}^{0},\mathsf{S}_{*})\simeq{\mathsf{F}}(\mathsf{x}^{0})-{\mathsf{F}}_{*}$.

We now provide an additional convergence theorem, analyzing the strongly convex case, which extends [21, Theorem 1] and [38, Theorem 3] to an arbitrary (not necessarily uniform) sampling and to the more general stepsize rule (1.5). The proof is still based on Proposition 4.6iii and is postponed to Appendix B.

Theorem 4.10**.**

Under the same assumptions of Theorem 4.9, let $\mu_{\mathsf{\Gamma}^{-1}}$ and $\sigma_{\mathsf{\Gamma}^{-1}}$ be the moduli of strong convexity of ${\mathsf{f}}$ and ${{\mathsf{h}}}$ respectively, in the norm $\lVert\cdot\rVert_{{\mathsf{\Gamma}}^{-1}}$, and suppose that $\mu_{\mathsf{\Gamma}^{-1}}+\sigma_{\mathsf{\Gamma}^{-1}}>0$ and that $\mathsf{S}_{*}=\{\mathsf{x}_{*}\}$. Then, for every $n\in\mathbb{N}$,

[TABLE]

where

[TABLE]

Remark 4.11**.**

Let $\gamma_{i}=\delta/\nu_{i}$ and ${\mathsf{V}}=\bigoplus_{i=1}^{m}\nu_{i}\mathsf{Id}_{i}$. Let $\mu_{{\mathsf{V}}}$ and $\sigma_{{\mathsf{V}}}$ be the moduli of strong convexity of ${\mathsf{f}}$ and ${{\mathsf{h}}}$ respectively, in the norm $\lVert\cdot\rVert_{{\mathsf{V}}}$. Then it is easy to see that $\mu_{\mathsf{\Gamma}^{-1}}=\delta\mu_{{\mathsf{V}}}$ and $\sigma_{\mathsf{\Gamma}^{-1}}=\delta\sigma_{{\mathsf{V}}}$. Moreover, as in Remark 2.1, one can also see that $\mu_{{\mathsf{V}}}\leq 1$. Then, (4.21) becomes

[TABLE]

One can check that the maximum of $\bar{\lambda}$ with respect to $\delta\in\left]0,2\right[$ is

[TABLE]

which is achieved at

[TABLE]

Note that if $\mu_{{\mathsf{V}}}<1$ (as is normally the case), then $\bar{\lambda}_{\mathrm{opt}}\in\left]0,1\right[$ and $\delta_{\mathrm{opt}}>1$.

Remark 4.12**.**

If $\mu$ and $\sigma$ are the moduli of strong convexity of ${\mathsf{f}}$ and ${{\mathsf{h}}}$ respectively in the original norm. Let, for every $i\in[m]$, $\gamma_{i}=\delta/\nu_{i}$ and set $\nu_{\max}=\max_{1\leq i\leq m}\nu_{i}$. Then $\mu_{{\mathsf{V}}}=\mu/\nu_{\max}$ and $\sigma_{{\mathsf{V}}}=\sigma/\nu_{\max}$ Therefore, the optimal stepsizes are achieved for

[TABLE]

and the corresponding rate in Theorem 4.10 becomes

[TABLE]

Remark 4.13**.**

Suppose that the block sampling is uniform, that is, $\mathsf{p}_{i}=\mathsf{p}$ for all $i\in[m]$ and let, for every $i\in[m]$, $\gamma_{i}=1/\nu_{i}$. Then $\delta=1$ and Theorem 4.10 reduce to

[TABLE]

This result was obtained in [38, Theorem 3], which is in turn a generalization of [21, Theorem 1], treating the serial case ($\mathsf{P}(\sum_{i=1}^{m}\varepsilon_{i}=1)=1$). Thus, Theorem 4.10 and the subsequent Remark 4.11 show that the rate in (4.26) can indeed be improved by choosing $\delta>1$.

4.3 Linear convergence under error bound conditions

In this section we analyze the convergence of Algorithm 1.1 under error bound conditions. We improve and simplify the results given in [23]. In the rest of the section we assume 1 and 2. Moreover, we let ${\mathsf{X}}\subset\mathsf{H}$, ${\mathsf{F}}={\mathsf{f}}+{{\mathsf{h}}}$, ${\mathsf{F}}_{*}=\inf{\mathsf{F}}$, and suppose $\mathsf{S}_{*}:=\operatorname*{\text{\rm argmin}}{\mathsf{F}}\neq\varnothing$.

We consider the following condition, which was studied in [8] in connection with the proximal gradient method and is known as Luo-Tseng error bound condition [22].

1. \rmEB

For some $c_{{\mathsf{X}},{\mathsf{\Gamma}}^{-1}}>0$, we have

[TABLE]

Remark 4.14**.**

1. (i)

Another popular error bound condition is that of the metric subregularity of the subdifferential. More precisely, $\partial^{{\mathsf{\Gamma}}^{-1}}{\mathsf{F}}$ is $2$-metrically subregular on ${\mathsf{X}}$ with respect to the metric $\lVert\cdot\rVert_{{\mathsf{\Gamma}}^{-1}}$[8, 15] if for some $\zeta_{{\mathsf{X}},{\mathsf{\Gamma}}^{-1}}>0$ the following holds

[TABLE] 2. (ii)

1 and (4.28) are equivalent if ${{\mathsf{h}}}=0$, since in that case $\text{\sf prox}_{{{\mathsf{h}}}}^{{\mathsf{\Gamma}}^{-1}}=\mathsf{Id}$ and $c_{{\mathsf{X}},{\mathsf{\Gamma}}^{-1}}=\zeta_{{\mathsf{X}},{\mathsf{\Gamma}}^{-1}}^{-1}$. 3. (iii)

Since $\partial^{{\mathsf{\Gamma}}^{-1}}{\mathsf{F}}(x)={\mathsf{\Gamma}}\partial{\mathsf{F}}(x)$ and $\lVert\cdot\rVert\geq\gamma_{\min}^{1/2}\lVert\cdot\rVert_{{\mathsf{\Gamma}}^{-1}}$, it follows that if for every $x\in{\mathsf{X}}$, $\mathrm{dist}(\mathsf{x},\mathsf{S}_{*})\leq\zeta^{-1}_{{\mathsf{X}},\mathsf{Id}}\mathrm{dist}({0},\partial{\mathsf{F}}(\mathsf{x}))$, then (4.28) holds with constant $\zeta_{{\mathsf{X}},{\mathsf{\Gamma}}^{-1}}=\gamma_{\min}\zeta_{{\mathsf{X}},\mathsf{Id}}$. 4. (iv)

[8, Theorem 3.5] yields that for any Hilbert norm $\lVert\cdot\rVert_{\mathsf{W}}$, $\lVert\mathsf{x}-\text{\sf prox}^{{\mathsf{W}}}_{{{\mathsf{h}}}}(\mathsf{x}-\nabla^{{\mathsf{W}}}{\mathsf{f}}(\mathsf{x}))\rVert_{{\mathsf{W}}}\leq\mathrm{dist}_{{\mathsf{W}}}({0},\partial^{{\mathsf{W}}}{\mathsf{F}}(\mathsf{x}))$. So, if 1 holds on ${\mathsf{X}}$, then (4.28) holds on ${\mathsf{X}}$ with $\zeta_{{\mathsf{X}},{\mathsf{\Gamma}}^{-1}}=c_{{\mathsf{X}},{\mathsf{\Gamma}}^{-1}}^{-1}$. In [8, Theorem 3.4-3.5] also the reverse implication was shown when ${\mathsf{f}}$ is Lipschitz smooth and ${\mathsf{X}}$ is a sublevel set of ${\mathsf{F}}$.

Remark 4.15**.**

In [8, Corollary 3.6] condition 1 was shown to be equivalent to the following quadratic growth condition (also called $2$-conditioning in [15])

[TABLE]

on every sublevel set ${\mathsf{X}}=\{\mathsf{x}\in\mathsf{H}\,|\,{\mathsf{F}}(\mathsf{x})-{\mathsf{F}}_{*}\leq r\}$. Moreover, the relationships between the constants are $c_{{\mathsf{X}},{\mathsf{\Gamma}}^{-1}}=(1+2/\alpha_{{\mathsf{X}},{\mathsf{\Gamma}}^{-1}})(1+L_{\lVert\cdot\rVert_{{\mathsf{\Gamma}}^{-1}}})$ and $\alpha_{{\mathsf{X}},{\mathsf{\Gamma}}^{-1}}<1/c_{{\mathsf{X}},{\mathsf{\Gamma}}^{-1}}$. Finally, if the quadratic growth condition (4.29) holds, then (4.28) holds on ${\mathsf{X}}$, with $\zeta_{{\mathsf{X}},{\mathsf{\Gamma}}^{-1}}=\alpha_{{\mathsf{X}},{\mathsf{\Gamma}}^{-1}}/2$.

We now analyze the convergence of Algorithm 1.1 under condition 1.

Theorem 4.16**.**

Under the assumptions of Theorem 4.9, suppose that $\mathsf{S}_{*}\neq\varnothing$ and that 1 holds on a set ${\mathsf{X}}$ such that {\mathsf{X}}\supset\big{\{}x^{n}\,|\,n\in\mathbb{N}\big{\}} $\mathsf{P}$-a.s. with $c_{{\mathsf{X}},{\mathsf{\Gamma}}^{-1}}>0$. Then,

[TABLE]

Moreover, there exists a random variable $x_{*}$ which takes values in $\mathsf{S}_{*}$ such that $x^{n}\to x_{*}$ $\mathsf{P}$-a.s. and \mathsf{E}[\lVert x^{n}-x_{*}\rVert_{{\mathsf{W}}}]=O\big{(}(1-\mathsf{p}_{\min}\min\{1,(2-\delta)/(2c_{{\mathsf{X}},{\mathsf{\Gamma}}^{-1}})\})^{n/2}\big{)}.

Proof.

Let $n\in\mathbb{N}$ and $\mathsf{x}\in\mathsf{S}_{*}$. Then, (4.8) with $\mu_{\mathsf{\Gamma}^{-1}}=\sigma_{\mathsf{\Gamma}^{-1}}=0$, yields

[TABLE]

Since (4.31) holds for all $\mathsf{x}\in\mathsf{S}_{*}$, using 1, (4.15), and Proposition 4.5, we have

[TABLE]

which can be equivalently written as

[TABLE]

Therefore,

[TABLE]

which gives (4.30). Now we set \rho=1-\mathsf{p}_{\min}\min\big{\{}1,(2-\delta)/(2c_{{\mathsf{X}},{\mathsf{\Gamma}}^{-1}})\big{\}} and $\theta=\mathsf{p}_{\min}(2-\delta)/2$. Then, Jensen inequality, (4.16), and (4.30) yield

[TABLE]

Therefore, since $\rho^{1/2}<1$, we have $\mathsf{E}[\sum_{n\in\mathbb{N}}\lVert x^{n}-x^{n+1}\rVert_{{\mathsf{W}}}]=\sum_{n\in\mathbb{N}}\mathsf{E}[\lVert x^{n}-x^{n+1}\rVert_{{\mathsf{W}}}]<+\infty$. Hence $\sum_{n\in\mathbb{N}}\lVert x^{n}-x^{n+1}\rVert_{{\mathsf{W}}}<+\infty$ $\mathsf{P}$-a.s., which means that $(x^{n})_{n\in\mathbb{N}}$ is a Cauchy sequence $\mathsf{P}$-a.s. Now, Theorem 4.9ii yields that there exists a random variable $x_{*}$ with values in $\mathsf{S}_{*}$ such that $x^{n}\rightharpoonup x_{*}$ $\mathsf{P}$-a.s. Therefore, $x^{n}\to x_{*}$ $\mathsf{P}$-a.s. Finally, let $n\in\mathbb{N}$. Then, for every $p\in\mathbb{N}$,

[TABLE]

Hence, letting $p\to+\infty$, we have $\mathsf{E}[\lVert x^{n}-x_{*}\rVert_{\mathsf{W}}]\leq\sum_{i=0}^{+\infty}\mathsf{E}[\lVert x^{n+i}-x^{n+i+1}\rVert_{\mathsf{W}}]$. Therefore, it follows from (4.33) that

[TABLE]

Remark 4.17**.**

1. (i)

The rate given in Theorem 4.16 matches the one given in [8, Theorem 3.2] for the deterministic case ($\mathsf{p}_{\min}=1$). 2. (ii)

In Theorem 4.16, the constant $c_{{\mathsf{X}},{\mathsf{\Gamma}}^{-1}}$ depends on the stepsizes $\gamma_{i}$’s which in turn depend on $\delta$ (usually $\gamma_{i}=\delta/\nu_{i}$ with $0<\delta<2$). Therefore, the optimal value of $\delta$ in the rate (4.30) can be determined after specifying the expression of $c_{{\mathsf{X}},{\mathsf{\Gamma}}^{-1}}$. We did so in the special application of Section 5.2. 3. (iii)

In [23, Definition 5.2], in relation to Algorithm 1.1 but with uniform block sampling and assuming 1 and $R_{{\mathsf{\Gamma}}^{-1}}(\mathsf{x}^{0}):=\sup_{\mathsf{x}\in\{{\mathsf{F}}\leq{\mathsf{F}}(\mathsf{x}^{0})\}}\mathrm{dist}_{{\mathsf{\Gamma}}^{-1}}(\mathsf{x},\mathsf{S}_{*})<+\infty$, the following error bound condition is considered

[TABLE]

for some constants $\kappa_{1,{\mathsf{X}},{\mathsf{\Gamma}}^{-1}}>0$ and $\kappa_{2,{\mathsf{X}},{\mathsf{\Gamma}}^{-1}}\geq 0$. The authors show several examples in which such condition is satisfied with ${\mathsf{X}}=\operatorname*{\text{\rm dom}}{{\mathsf{h}}}$ and possibly $\kappa_{2,{\mathsf{X}},{\mathsf{\Gamma}}^{-1}}>0$. The above error bound looks more general then 1. However, for the purpose of analyzing Algorithm 1.1 and under the assumptions considered in [23] this is not the case. Indeed, in [23, equation (3.11)] it was shown that the algorithm is descending almost surely222Alternatively, note that 1 implies 2 which in turn, in view of Proposition 4.7, ensures the descending property., so $\{x^{n}\,|\,n\in\mathbb{N}\}\subset\{{\mathsf{F}}\leq{\mathsf{F}}(\mathsf{x}^{0})\}$ $\mathsf{P}$-a.s. Therefore, since $\sup_{\mathsf{x}\in\{{\mathsf{F}}\leq{\mathsf{F}}(\mathsf{x}^{0})\}}\mathrm{dist}_{{\mathsf{\Gamma}}^{-1}}(\mathsf{x},\mathsf{S}_{*})=R_{{\mathsf{\Gamma}}^{-1}}(\mathsf{x}^{0})<+\infty$, if (4.34) holds on a set ${\mathsf{X}}$ containing $\mathsf{P}$-a.s. the set $\{x^{n}\,|\,n\in\mathbb{N}\}$, then 1 holds on ${\mathsf{X}}^{\prime}:={\mathsf{X}}\cap\{{\mathsf{F}}\leq{\mathsf{F}}(\mathsf{x}^{0})\}\supset\{x^{n}\,|\,n\in\mathbb{N}\}$ $\mathsf{P}$-a.s. with $c_{{\mathsf{X}}^{\prime},{\mathsf{\Gamma}}^{-1}}:=\kappa_{1,{\mathsf{X}},{\mathsf{\Gamma}}^{-1}}+\kappa_{2,{\mathsf{X}},{\mathsf{\Gamma}}^{-1}}R_{{\mathsf{\Gamma}}^{-1}}(\mathsf{x}^{0})^{2}$. Thus, Theorem 4.16 applies accordingly. Moreover, [23, Theorem 5.5] gives the linear rate

[TABLE]

Then, we have $1+\bar{c}\geq(3+4c_{{\mathsf{X}}^{\prime},{\mathsf{\Gamma}}^{-1}})/\mathsf{p}$ and hence

[TABLE]

This shows that Theorem 4.16 improves the rate in [23, Theorem 5.5]. Moreover, the analysis given here, relying on Proposition 4.6, relaxes the assumptions and is significantly simpler. 4. (iv)

It follows from [23, Theorem 6.8] that if ${\mathsf{f}}$ is a quadratic function and ${{\mathsf{h}}}$ is an indicator function of a polyhedral set, then (4.34) is satisfied on $\operatorname*{\text{\rm dom}}{{\mathsf{h}}}$. Therefore, if $\operatorname*{\text{\rm dom}}{{\mathsf{h}}}$ is bounded, then 1 holds on ${\mathsf{X}}=\operatorname*{\text{\rm dom}}{{\mathsf{h}}}$ with $c_{{\mathsf{X}},{\mathsf{\Gamma}}^{-1}}:=\kappa_{1,{\mathsf{X}},{\mathsf{\Gamma}}^{-1}}+\kappa_{2,{\mathsf{X}},{\mathsf{\Gamma}}^{-1}}\operatorname*{\text{\rm diam}}^{2}_{{\mathsf{\Gamma}}^{-1}}(\operatorname*{\text{\rm dom}}{{\mathsf{h}}})$ and Theorem 4.16 can be applied, since $\{x^{n}\,|\,n\in\mathbb{N}\}\subset\operatorname*{\text{\rm dom}}{{\mathsf{h}}}$. 5. (v)

Several works address the convergence of random coordinate descent methods under error bound conditions. We mention [16] which considers a serial sampling and stepsizes set according to the global Lipschitz constant of $\nabla{\mathsf{f}}$ and [14], which analyzes restarting procedures for accelerated and parallel coordinate descent methods using assumptions 1 and (4.29).

Remark 4.18**.**

Often error bound conditions or quadratic growth conditions are satisfied when ${\mathsf{X}}$ is a sublevel set (see Remark 4.15). So, in such scenarios, apart when $\operatorname*{\text{\rm dom}}{{\mathsf{h}}}$ is a sublevel set of ${\mathsf{F}}$, in order to fulfill the assumption ${\mathsf{X}}\supset\{x^{n}\,|\,n\in\mathbb{N}\}$ $\mathsf{P}$-a.s. in Theorem 4.16, it is desirable for Algorithm 1.1 to be (a.s.) descending. This occurs if condition 2 holds (Proposition 4.7), whereas, in general, 1 does not guarantee any such descending property. However, especially when $\eta\ll m$, condition 2 may be much more restrictive than 1, thus leading to a significant reduction of the stepsizes, which ultimately slows down the convergence. The next result shows that Algorithm 1.1 can be slightly modified so to ensure the descending property while keeping the validity of Theorem 4.16.

Theorem 4.19**.**

Let 1–1 be satisfied. Let $(\nu_{i})_{1\leq i\leq m}\in\mathbb{R}^{m}_{++}$ and suppose that 1 holds. Suppose in addition that $\mathsf{S}_{*}\neq\varnothing$, and that 1 holds on the set ${\mathsf{X}}=\{{\mathsf{F}}\leq{\mathsf{F}}(\mathsf{x}^{0})\}$ with $c_{{\mathsf{X}},{\mathsf{\Gamma}}^{-1}}>0$. Let $(x^{n})_{n\in\mathbb{N}}$ be generated by the following variation of Algorithm 1.1

[TABLE]

Then the conclusions of Theorem 4.16 still hold.

Proof.

It follows from the definition of $x^{n+1}$ that ${\mathsf{F}}(x^{n+1})\leq{\mathsf{F}}(x^{n})$. Therefore ${\mathsf{X}}\supset\{x^{n}\,|\,n\in\mathbb{N}\}$. Recalling (4.2) and (4.3), algorithm (4.35) can be alternatively written as

[TABLE]

and we have ${\mathsf{F}}(x^{n+1})\leq{\mathsf{F}}(\tilde{x}^{n+1})$. Then we can essentially repeat the argument in the proof of Theorem 4.16. First we note that (4.8) and hence (4.31) holds with $x^{n+1}$ replaced by $\tilde{x}^{n+1}$. This follows from the definition of $\tilde{x}^{n+1}$. Moreover, also (4.13) holds with $x^{n+1}$ replaced by $\tilde{x}^{n+1}$ and hence we derive (with $\mathsf{x}=x^{n}$ and $\sigma_{\mathsf{\Gamma}^{-1}}=0$) that

[TABLE]

Then, we have

[TABLE]

and hence, since ${\mathsf{F}}(x^{n+1})\leq{\mathsf{F}}(\tilde{x}^{n+1})$, (4.32) still holds (for the new definition of $x^{n+1}$). Thus, (4.30) follows. As for the second part of the statement, we note that, since ${\mathsf{F}}(x^{n+1})\leq{\mathsf{F}}(\tilde{x}^{n+1})$, by (4.37), we have $(2-\delta)\lVert\bar{x}^{n+1}-x^{n}\rVert_{{\mathsf{\Gamma}}^{-1}}^{2}\leq(2/\mathsf{p}_{\min})\mathsf{E}[{\mathsf{F}}(x^{n})-{\mathsf{F}}(x^{n+1})\,|\,\mathfrak{E}_{n-1}]$. Moreover, it follows from the definitions of $x^{n+1}$ and $\tilde{x}^{n+1}$ in algorithm (4.36) that

[TABLE]

and hence, by Proposition 4.5, we have

[TABLE]

In the end (4.16) with $\sigma_{{\mathsf{\Gamma}}^{-1}}=0$ still holds and the proof can continue as in that of Theorem 4.16. ∎

5 Applications

In this section we show some relevant optimization problems for which the theoretical analysis of Algorithm 1.1 can be particularly useful.

5.1 The Lasso problem

Many papers study the convergence of coordinate descent methods for the Lasso problem and recent works prove linear convergence (see e.g., [16, 23, 25]). In [16] a random serial update of blocks is considered while in [25] the general framework of feasible descend methods is analyzed which include (nonrandom) cyclic coordinate methods. In the following we discuss our contribution comparing with [23]. Let ${\mathsf{A}}\in\mathbb{R}^{p\times m}$ and $\mathsf{b}\in\mathbb{R}^{p}$. We consider the problem

[TABLE]

We denote by $\mathsf{a}^{i}$ and $\mathsf{a}_{k}$ the $i$-th column and $k$-th row of ${\mathsf{A}}$ respectively. Since

[TABLE]

1 holds and $\eta=\max_{1\leq k\leq p}\mathrm{card}(\mathrm{spt}(\mathsf{a}_{k}))$. Moreover, since $\nabla_{i}{\mathsf{f}}(\mathsf{x})=\langle\mathsf{a}^{i},{\mathsf{A}}\mathsf{x}-\mathsf{b}\rangle$ and recalling Remark 3.2i, conditions 1 and 2 are satisfied with $L_{i}=\lVert\mathsf{a}^{i}\rVert^{2}$ and $L^{(k)}=\lVert\mathsf{a}_{k}\rVert^{2}$ respectively. Then, Algorithm 1.1 (assuming that each block is made of one coordinate only) writes as

[TABLE]

where the soft thresholding operator $\mathrm{soft}_{\gamma_{i}\lambda}$ is defined as $\mathrm{soft}_{\gamma_{i}\lambda}(t)=\mathrm{sign}(t)\max\{0,\lvert t\rvert-\gamma_{i}\lambda\}$ and $(\mathsf{e}_{i})_{1\leq i\leq m}$ is the canonical basis of $\mathbb{R}^{m}$. Now, define $u^{n}={\mathsf{A}}x^{n}-\mathsf{b}$. Then multiplying the equation in (5.2) by ${\mathsf{A}}$ and subtracting $\mathsf{b}$ by both terms, the algorithm is equivalently written as

[TABLE]

showing that each iteration costs $O(p\tau_{\max})$ multiplications, where $\tau_{\max}$ is the maximum number of block updates per iteration. We now address the determination of the smoothness parameters $(\nu_{i})_{1\leq i\leq m}$. We first give a general rule which holds for any arbitrary sampling. Recalling Theorem 3.1iii and Remark 3.2i and noting that $\{k|i\in I_{k}\}=\{k|i\in\mathrm{spt}(\mathsf{a}_{k})\}=\mathrm{spt}(\mathsf{a}^{i})$, if we set $\nu_{i}=\sum_{k\in\mathrm{spt}(\mathsf{a}^{i})}\lVert\mathsf{a}_{k}\rVert^{2}$, then 1 and hence 2 holds. This choice was considered in [23]. Moreover, according to the discussion at the beginning of Section 3.2 other options for satisfying 2 are $\nu_{i}=\min\{\eta,\tau_{\max}\}\lVert\mathsf{a}^{i}\rVert^{2}$ or $\nu_{i}=\sum_{k\in\mathrm{spt}(\mathsf{a}^{i})}\min\{\mathrm{card}(\mathrm{spt}(\mathsf{a}_{k})),\tau_{\max}\}(\mathsf{a}_{k}^{i})^{2}$. This latter choice is better than the second one and, if we assume that the nonzero entries of $A$ are about of the same magnitude, it is also better than the first one. Next, we face the special case of the $\tau$-nice sampling which allows to reduce the $\nu_{i}$’s while satisfying 1. Recalling the corresponding discussion in Section 3.2, we have the following alternatives: (1) set, for every $i\in[m]$, $\nu_{i}=(1+(\tau-1)(\eta-1)/(m-1))\lVert\mathsf{a}^{i}\rVert^{2}$; (2) set for every $i\in[m]$, $\nu_{i}=\sum_{k\in\mathrm{spt}(\mathsf{a}^{i})}(1+(\tau-1)(\mathrm{card}(\mathrm{spt}(\mathsf{a}_{k}))-1)/(m-1))(\mathsf{a}_{k}^{i})^{2}$. Finally, we make few remarks on the convergence properties of algorithm (5.3). Since the objective function in (5.1) satisfies a quadratic growth condition on its sublevel sets [15, Example 3.8], then Remark 4.15, Remark 4.18, and Theorem 4.16, yield linear convergence of algorithm (5.3) provided that 2 holds. Whereas Theorem 4.19 ensures that if we modify algorithm (5.3) so that we accept the next iterate $x^{n+1}$ only if $\lVert u^{n+1}\rVert^{2}+2\lambda\lVert x^{n+1}\rVert_{1}\leq\lVert u^{n}\rVert^{2}+2\lambda\lVert x^{n}\rVert_{1}$, then the resulting algorithm converges linearly under condition 1. If the violation of the monotonicity condition above occurs few times along all the iterations, this modification does not increase much the computational cost of the algorithm (see also Section 6). We stress that both Theorem 4.16 and Theorem 4.19 ensure also almost sure and linear convergence in mean of the iterates of (5.2). This latter result is new and is especially relevant in this context, since the iterates carry sparsity information.

5.2 Computing the minimal norm solution of a linear system

Let ${\mathsf{A}}\in\mathbb{R}^{m\times p}$ and $\mathsf{b}\in R({\mathsf{A}})$ (the range of ${\mathsf{A}}$). Let us consider the problem

[TABLE]

Here, we denote by $\mathsf{a}_{i}\in\mathbb{R}^{p}$ and $\mathsf{a}^{k}\in\mathbb{R}^{m}$ the $i$-th row and the $k$-th column of ${\mathsf{A}}$. The dual problem is

[TABLE]

which is a smooth convex optimization problem. Moreover, if $\mathsf{x}_{*}$ is the solution of (5.4) and $\mathsf{x}={\mathsf{A}}^{\top}\mathsf{u}$ (the primal-dual relationship), then we have

[TABLE]

Then, the dual problem is clearly of the form (1.1), with ${{\mathsf{h}}}=0$, and 1 and 1 are satisfied, assuming that each block is made of one coordinate only, with $L_{i}=\lVert\mathsf{a}_{i}\rVert^{2}$ and $\eta=\max_{1\leq k\leq p}\mathrm{card}(\mathrm{spt}(\mathsf{a}^{k}))$. So, Algorithm 1.1 applied to (5.5), turns into

[TABLE]

Now, setting $x^{n}={\mathsf{A}}^{\top}u^{n}$ and multiplying the above equality by ${\mathsf{A}}^{\top}$, we have

[TABLE]

Since, $\mathsf{b}\in R({\mathsf{A}})$, it is easy to see, through a singular value decomposition of ${\mathsf{A}}$, that, for every $\mathsf{u}\in\mathbb{R}^{m}$, $\sigma^{2}_{\min}({\mathsf{A}})\,\mathrm{dist}(\mathsf{u},\operatorname*{\text{\rm argmin}}\mathcal{D})\leq\lVert\nabla\mathcal{D}(\mathsf{u})\rVert$ (where $\sigma_{\min}({\mathsf{A}})$ is the minimum singular value of ${\mathsf{A}}$) [15, Example 3.6]. So, in view of Remark 4.14ii-iii, 1 is satisfied on the entire space with constant $c_{\mathbb{R}^{m},{\mathsf{\Gamma}}^{-1}}=(\gamma_{\min}\sigma^{2}_{\min}({\mathsf{A}}))^{-1}$. Therefore, if, for every $i\in[m]$, $\gamma_{i}=\delta/(\beta_{1,i}\lVert\mathsf{a}_{i}\rVert^{2})$ with $0<\delta<2$, Theorem 4.16 and (5.6) ensure the linear convergence of the iterates $x^{n}$’s towards the solution of (5.4) with rate \big{(}1-\mathsf{p}_{\min}\min\big{\{}1,\gamma_{\min}\sigma^{2}_{\min}({\mathsf{A}})(2-\delta)/2\big{\}}\big{)}^{1/2}. We remark that (5.7) is nothing but a stochastic gradient descent algorithm on the problem

[TABLE]

Since $\lVert{\mathsf{A}}\mathsf{x}-\mathsf{b}\rVert^{2}\leq\lVert{\mathsf{A}}\rVert^{2}\lVert\mathsf{x}-\mathsf{x}_{*}\rVert^{2}$, we have then showed the linear convergence rate

[TABLE]

of the stochastic gradient descent with arbitrary and possibly variable batch size for least squares problems. This also shows that the best rate is achieved for $\delta=1$. We finally note that in the serial case, that is, if for every $n\in\mathbb{N}$ $\mathrm{spt}(\varepsilon^{n})=\{i_{n}\}$, multiplying equation (5.7) by $\mathsf{a}_{i_{n}}^{\top}$, we have

[TABLE]

Therefore, since in this case $\beta_{1,i}=\beta_{2}=1$, we can chose the stepsizes such that $\gamma_{i}\lVert\mathsf{a}_{i}\rVert^{2}=1$ (so that $\delta=1$) and hence $x^{n+1}$ is a solution of the $i_{n}$-th equation of the linear system ${\mathsf{A}}\mathsf{x}=\mathsf{b}$. Moreover, $x^{n+1}$ is the projection of $x^{n}$ onto the affine space defined by the equation $\mathsf{a}_{i_{n}}\mathsf{x}=\mathsf{b}_{i_{n}}$ [41]. Thus, this method is nothing but the randomized Kaczmarz method [37] and we proved linear convergence for general probabilities $\mathsf{p}_{i}$’s, although the constants we derive are not optimal (see [19, 37, 41]).

5.3 Regularized empirical risk minimization

Let $\mathsf{H}$ be a separable real Hilbert space. Regularized empirical risk estimation solves the following optimization problem

[TABLE]

where $(\mathsf{x}_{i},\mathsf{y}_{i})_{1\leq i\leq m}$ is the training set (input-output pairs), $\ell\colon\mathcal{Y}\times\mathbb{R}\to\mathbb{R}_{+}$, $\mathcal{Y}\subset\mathbb{R}$, is the loss function, which is convex in the second variable, and $\lambda>0$ is a regularization parameter. The dual problem of (5.8) is

[TABLE]

where $\ell^{*}(\mathsf{y}_{i},\cdot)$ is the Fenchel conjugate of $\ell(\mathsf{y}_{i},\cdot)$ and $\mathsf{K}={\mathsf{X}}{\mathsf{X}}^{\top}\in\mathbb{R}^{m\times m}$ is the Gram matrix of $(\mathsf{x}_{i})_{1\leq i\leq m}$. Moreover, the solutions $(\bar{\mathsf{w}},\bar{\mathsf{u}})$ of the primal and dual problems are characterized by the following KKT conditions

[TABLE]

where $\partial\ell(\mathsf{y}_{i},\cdot)$ is the subdifferential of $\ell(\mathsf{y}_{i},\cdot)$. Note also that the first of (5.10) gives the link between the dual and the primal variable and, if $\mathsf{w}={\mathsf{X}}^{\top}\mathsf{u}$, then it holds $(1/2)\lVert\mathsf{w}-\bar{\mathsf{w}}\rVert^{2}\leq\mathcal{D}(\mathsf{u})-\inf\mathcal{D}$. Now, the dual problem (5.9) is of the form (1.1) and hence Algorithm 1.1 can be applied. The following examples give implementation details for two specific losses.

Example 5.1** (Ridge regression).**

The least squares loss is $\ell(s,t)=(1/2)\lvert s-t\rvert^{2}$. Then $\ell^{*}(s,r)=(1/2)r^{2}+rs$ and, in this case, (5.9) reduces to

[TABLE]

which is strongly convex with modulus $\lambda m$ and has solution $\bar{\mathsf{u}}=(\mathsf{K}+m\lambda\mathsf{Id})^{-1}\mathsf{y}$. Since $\mathcal{D}$ is smooth and $\nabla_{i}\mathcal{D}(\mathsf{u})=\mathsf{e}_{i}^{\top}(\mathsf{K}+\lambda m\mathsf{Id})\mathsf{u}-\mathsf{y}_{i}$, conditions 1 and 1 hold with $L_{i}=\lVert\mathsf{x}_{i}\rVert^{2}+\lambda m=\mathsf{K}_{i,i}+\lambda m$ and Algorithm 1.1 (with ${{\mathsf{h}}}=0$) becomes

[TABLE]

Moreover, multiplying (5.11) by ${\mathsf{X}}^{\top}$, defining $w^{n}={\mathsf{X}}^{\top}u^{n}$, and recalling that $\mathsf{K}={\mathsf{X}}{\mathsf{X}}^{\top}$, we have

[TABLE]

Note that, since the dual problem is strongly convex with modulus $\lambda m$, then it follows from Theorem 4.10, Remark 4.12, and Theorem C.1i that, setting, for every $i\in[m]$, $\nu_{i}\geq\beta_{1,i}(\mathsf{K}_{i,i}+\lambda m)$ and $\gamma_{i}=1/\nu_{i}$, we have

[TABLE]

Now, we compare algorithm (5.12) with the stochastic gradient descent on problem (5.8). Assume that $\mathsf{P}(\sum_{i=1}^{m}\varepsilon^{n}_{i}=\tau)=1$ for some $\tau\in[m]$. Then, $\mathsf{p}_{\min}=\tau/m$ and we can take $\zeta_{i}\leq(\mathsf{K}_{i,i}+\lambda m)^{-1}$, and set $\nu_{i}=\tau/\zeta_{i}$ and $\gamma_{i}=1/\nu_{i}$, so that algorithm (5.12) turns into

[TABLE]

If we apply stochastic gradient descent with batch size $\tau\in[m]$ and stepsize $\zeta>0$ directly on the primal problem (5.8) (multiplied by $\lambda m$), and recalling that $w^{n}=\sum_{i=1}^{m}u^{n}_{i}\mathsf{x}_{i}$, we have

[TABLE]

Then, comparing (5.13) and (5.14) we see that, provided that $\zeta_{i}=\zeta$ for every $i\in[m]$, they only differ for the replacement $(1/m)\sum_{i=1}^{m}u^{n}_{i}\mathsf{x}_{i}\ \leftrightarrow\ (1/\tau)\sum_{i=1}^{m}\varepsilon^{n}_{i}u^{n}_{i}\mathsf{x}_{i}$. We stress that the stepsize $\zeta$ in the stochastic gradient descent algorithm (5.14) is normally set according to the spectral norm of $\mathsf{K}+\lambda m\mathsf{Id}$, which may be difficult to compute. On the contrary in algorithm (5.13) the stepsizes $\zeta_{i}$’s are simply set as $\zeta_{i}\leq 1/(\mathsf{K}_{i,i}+\lambda m)$, so they allow possibly much longer steps and also do not require any SVD computation.

Example 5.2** (Support vector machines).**

The hinge loss is $\ell(s,t)=(1-st)_{+}$. Then we have $\ell^{*}(s,r)=r+\iota_{[0,1]}(sr)$ and the dual problem (5.9) is

[TABLE]

Then Algorithm 1.1 on the dual turns into a parallel random block-coordinate projected gradient descent method. Moreover, it follows from Remark 4.17iv that the objective in (5.15) satisfies 1 on its domain. Therefore, it follows from Theorem 4.16, Theorem 3.1i, and Theorem C.1ii that $\mathsf{E}[\mathcal{P}(w^{n})]-\inf\mathcal{P}$ converges linearly to zero, provided that, for all $i\in[m]$, $\nu_{i}\geq\beta_{1,i}\mathsf{K}_{ii}$ and $\gamma_{i}<2/\nu_{i}$.

6 Numerical Experiments

In this section we consider a Lasso problem, that is,

[TABLE]

where ${\mathsf{A}}\in\mathbb{R}^{p\times m}$ is generated with random entries uniformly distributed in $[-1,1]$ so that each row is sparse and $\mathsf{b}={\mathsf{A}}\bar{\mathsf{x}}+0.06\cdot\alpha$ with $\alpha\sim N(0,\mathsf{Id}_{p})$ and $\bar{x}$ a sparse vector in $\mathbb{R}^{m}$. We implement Algorithm 1.1 with $\gamma_{i}=\delta/\nu_{i}$ ($0<\delta<2$) and a $\tau$-nice uniform sampling, as described in Section 5.1. We present two experiments. The first compares conditions 1 and 2 for the determination of the stepsizes $\gamma_{i}$. The second one investigates the role played by $\delta$. In all the experiments we empirically checked that the algorithm is essentially descending in the sense that during the iterations there are very few violations of the descent property and with low magnitude. So, since the objective function in (6.1) satisfies 1 on the sublevel sets, in virtue of Theorem 4.19, linear convergence holds.

Condition 1 vs 2 and the effectiveness of the parallel strategy.

We compare the conditions 1 and 2 for the stepsizes selection and we checked the critical role played by 1 for the effectiveness of the parallel strategy on problems with sparse structure. Here we set $\delta=1$. In Figure 1, Algo1 uses smoothness parameters specifically designed for the $\tau$-nice sampling, that is, $\nu_{i}=\beta_{1}\lVert\mathsf{a}^{i}\rVert^{2}$ with $\beta_{1}=1+(\tau-1)(\eta-1)/(m-1)$ (making 1 satisfied), while Algo2 uses a more conservative choice for the smoothness parameters which is valid for any sampling updating a maximum of $\tau$ blocks per iteration, that is $\nu_{i}=\beta_{2}\lVert\mathsf{a}^{i}\rVert^{2}$ with $\beta_{2}=\min\{\tau,\eta\}$ (making also 2 satisfied).

In the left diagram, we considered a large scale setting with $\eta\ll m$. In that case $\beta_{1}$ may be much smaller than $\beta_{2}$ leading to significantly larger stepsizes. Moreover, and more importantly, we note that as long as $\tau$ is small enough the behavior of Algo1 does not depend on $\tau$ (indeed $\tau=10,50,100$ perform equally well), whereas this is not true for Algo2. This feature of 1, first noted in [34], has been already discussed after Theorem 4.9 and is at the basis of the effectiveness of the parallel strategy. Indeed in the small-$\tau$ regime described above, the various versions of Algo1 depicted in Figure 1(left) have the same total computation cost ($\bar{n}m$ block-coordinate updates), but the parallel implementation on $\tau$ cores is $\tau$ times faster than the serial one ($\tau=1$). Finally, in the right diagram of Figure 1 we show a scenario in which $\eta/m$ is larger (the problem is less sparse). In such situation we see that the difference between the two stepsize selection criteria is less evident for $\tau\geq 50$. Moreover, Algo1 is more sensitive to $\tau$ (compare $\tau=10$ and $\tau=50$), so that the benefit of the parallel scheme is reduced.

The effect of $\delta>1$

Here we study the effect of over-relaxing the stepsizes, meaning choosing $\delta>1$. We compare Algorithm 1.1 with $\delta=1$ and several choices of $\delta>1$. Figure 2 considers different scenarios for the degree of separability $\eta$ of $f$. In those cases we see that choosing $\delta>1$ usually speeds up the convergence, depending on the parameter $\eta$ of partial separability of $f$ and the number $\tau$ of parallel block updates. This fact seems not to occur when both $\eta/m$ and $\tau/m$ are very small.

Appendix A Structured Lipschitz smoothness

In this section we discuss the Lipschitz smoothness properties of ${\mathsf{f}}$ under the hypotheses 1 and 1 and we prove Theorem 3.1. Most of the results presented in this section are basically given in [34]. However, here they are rephrased in our notation and extended to our more general assumptions.

Proposition A.1**.**

Let ${\mathsf{f}}\colon\mathsf{H}\to\mathbb{R}$ be a convex function satisfying assumptions 1 and 1. Let $I$ be a nonempty subset of $[m]$ and let $(q_{i})_{i\in I}\in\mathbb{R}_{++}$ be such that $\sum_{i\in I\cap I_{k}}q_{i}\leq 1$, for every $k\in[p]$. Then for every $\mathsf{x}$ and $\mathsf{y}\in\mathsf{H}$ such that $\mathrm{spt}(\mathsf{x}-\mathsf{y})\subset I$, we have

[TABLE]

Proof.

Let $\mathsf{v}=\mathsf{y}-\mathsf{x}$ and, for every $k\in[p]$, set $\mathsf{z}^{(k)}=\sum_{i=1}^{m}\mathsf{U}_{k,i}\mathsf{x}_{i}$. Then

[TABLE]

Now, for every $k\in[p]$, we have

[TABLE]

Therefore, using the convexity of each ${\mathsf{g}}_{k}$ we have

[TABLE]

It follows from the definition of $I_{k}$ that $\sum_{i\in I\setminus I_{k}}q_{i}{\mathsf{g}}_{k}(\mathsf{z}^{(k)}+q_{i}^{-1}\mathsf{U}_{k,i}\mathsf{v}_{i})=\sum_{i\in I\setminus I_{k}}q_{i}{\mathsf{g}}_{k}(\mathsf{z}^{(k)})$. Hence, switching the order of summation, and using the fact that ${\mathsf{f}}(\mathsf{x}_{1},\dots,\mathsf{x}_{i-1},\cdot,\mathsf{x}_{i+1},\dots,\mathsf{x}_{m})$ is Lipschitz smooth with constant $L_{i}$, we have

[TABLE]

Corollary A.2**.**

Let ${\mathsf{f}}\colon\mathsf{H}\to\mathbb{R}$ be a convex function satisfying 1 and 1. Let $\eta=\max_{1\leq k\leq m}\mathrm{card}(I_{k})$, $(\gamma_{i})_{1\leq i\leq m}\in\mathbb{R}^{m}_{++}$, and $(q_{i})_{1\leq i\leq m}\in\mathbb{R}^{m}_{++}$ be such that, for every $k\in[p]$, $\sum_{i\in I_{k}}q_{i}\leq 1$. Let $\mathsf{\Lambda}=\bigoplus_{i=1}^{m}L_{i}\mathsf{Id}_{i}$, $\mathsf{\Gamma}=\bigoplus_{i=1}^{m}\gamma_{i}\mathsf{Id}_{i}$, and $\mathsf{Q}=\bigoplus_{i=1}^{m}q_{i}\mathsf{Id}_{i}$. Then, the function ${\mathsf{f}}$ is Lipschitz smooth

1. (i)

in the metric $\lVert\cdot\rVert_{{\mathsf{W}}}$ defined by ${\mathsf{W}}=\mathsf{\Lambda}\mathsf{Q}^{-1}$ with constant $1$; 2. (ii)

in the metric $\lVert\cdot\rVert_{\mathsf{\Gamma}^{-1}}$, with constant333 This constant is $\leq\delta\eta/\min_{1\leq i\leq m}\beta_{1,i}$ if the $\gamma_{i}$’s are set according to (1.5) with the $\nu_{i}$’s as in Theorem 3.1i. * $\max_{1\leq k\leq p}\sum_{i\in I_{k}}\gamma_{i}L_{i}$.* 3. (iii)

in the (original) metric of $\mathsf{H}$ with constant $\max_{1\leq k\leq p}\sum_{i\in I_{k}}L_{i}$; 4. (iv)

in the metric $\lVert\cdot\rVert_{\mathsf{\Lambda}}$, with constant $\eta$.

Proof.

i: It follows from Proposition A.1 with $I=[m]$ and noting that $\langle\nabla{\mathsf{f}}(\mathsf{x}),\mathsf{y}-\mathsf{x}\rangle=\langle\nabla^{\mathsf{W}}{\mathsf{f}}(\mathsf{x}),\mathsf{y}-\mathsf{x}\rangle_{\mathsf{W}}$ and then invoking the characterization of the Lipschitz continuity of the gradient troughout the descent lemma (see [1, Theorem 18.15(iii)]).

ii: It follows from (A.1) by choosing $I=[m]$, $q_{i}=\gamma_{i}L_{i}/(\max_{1\leq k\leq p}\sum_{j\in I_{k}}\gamma_{j}L_{j})$, and noting that $\langle\nabla{\mathsf{f}}(\mathsf{x}),\mathsf{y}-\mathsf{x}\rangle=\langle\nabla^{\mathsf{\Gamma}^{-1}}{\mathsf{f}}(\mathsf{x}),\mathsf{y}-\mathsf{x}\rangle_{\mathsf{\Gamma}^{-1}}$ and then invoking [1, Theorem 18.15(iii)].

iii: It follows from ii with $\gamma_{i}=1$.

iv: It follows from ii with $\gamma_{i}=1/L_{i}$. ∎

Remark A.3**.**

If $\eta=m$ (${\mathsf{f}}$ is not partially separable), Corollary A.2-iii-iv establishes that

[TABLE]

We show that the above bounds are tight. Indeed, if we consider ${\mathsf{f}}(\mathsf{x})=(1/2)\lVert{\mathsf{A}}\mathsf{x}-\mathsf{b}\rVert_{2}^{2}$, where ${\mathsf{A}}\in\mathbb{R}^{n\times m}$ and $\mathsf{b}\in\mathbb{R}^{m}$, then we have $L_{i}=\lVert\mathsf{a}_{i}\rVert^{2}$ (where $\mathsf{a}_{i}$ is the $i$-th column of ${\mathsf{A}}$), so that $\sum_{i=1}^{m}L_{i}=\lVert{\mathsf{A}}\rVert_{F}^{2}$. Instead, since $\nabla{\mathsf{f}}(\mathsf{x})={\mathsf{A}}^{*}({\mathsf{A}}\mathsf{x}-\mathsf{b})$, the Lipschitz constant of $\nabla{\mathsf{f}}$ is $\lVert{\mathsf{A}}\rVert^{2}$. It is well-known that if ${\mathsf{A}}$ is rank one, then $\lVert{\mathsf{A}}\rVert^{2}=\lVert{\mathsf{A}}\rVert_{F}^{2}$, so in this case the Lipschitz constant of $\nabla{\mathsf{f}}$ is exactly $\sum_{i=1}^{m}L_{i}$. Moreover, if in addition the columns of ${\mathsf{A}}$ have the same norm, then $L=\sum_{j=1}^{m}L_{j}=mL_{i}$ and hence $L_{\lVert\cdot\rVert_{\mathsf{\Lambda}}}=m$. We finally note that if ${\mathsf{A}}$ is an orthonormal matrix, then $\lVert{\mathsf{A}}\rVert^{2}=1$ and hence $1=L_{i}=L=L_{\lVert\cdot\rVert_{\mathsf{\Lambda}}}<\sum_{i=1}^{m}L_{i}=m$.

Corollary A.4**.**

Let ${\mathsf{f}}\colon\mathsf{H}\to\mathbb{R}$ be a function satisfying 1 and 1. Let $\epsilon\in\{0,1\}^{m}$ and $\mathsf{x},\mathsf{v}\in\mathsf{H}$. Then,

[TABLE]

Proof.

It follows from Proposition A.1 with $\mathsf{y}=\mathsf{x}+\epsilon\odot\mathsf{v}$, $I=\mathrm{spt}(\epsilon)$, and q_{i}=1/\big{(}\max_{1\leq k\leq p}\mathrm{card}(I\cap I_{k})\big{)}=1/\big{(}\max_{1\leq k\leq p}\sum_{i\in I_{k}}\epsilon_{i}\big{)}. ∎

Remark A.5**.**

Most of the above results, appears in [34] for the special case that $\mathsf{U}_{k,i}=\mathsf{J}_{i}$ for $i\in I_{k}$ and $\mathsf{U}_{k,i}=0$ for $i\notin I_{k}$. In particular, see [34, Theorem 8].

Proposition A.6**.**

Let ${\mathsf{f}}\colon\mathsf{H}\to\mathbb{R}$ be a function satisfying 1 and suppose that, for every $k\in[p]$, ${\mathsf{g}}_{k}$ is $L^{(k)}$-Lipschitz smooth. Set for every $i\in[m]$, $\tilde{L}_{i}=\lVert\sum_{k=1}^{p}L^{(k)}\mathsf{U}_{k,i}^{\top}\mathsf{U}_{k,i}\rVert$. Then the following holds.

1. (i)

${\mathsf{f}}$* is Lipschitz smooth with constant $\lVert\sum_{k=1}^{p}L^{(k)}U_{k}^{\top}U_{k}\rVert$ in the original metric of $\mathsf{H}$;444 In [4, Corollary 5.11] the worse constant $\sum_{k=1}^{p}L^{(k)}\lVert\mathsf{U}_{k}\mathsf{U}_{k}^{\top}\rVert$ was considered.* 2. (ii)

${\mathsf{f}}$* satisfies assumption 1 with $L_{i}=\tilde{L}_{i}$;* 3. (iii)

Suppose that, for every $k\in[p]$ and for every $i,j\in[m]$, $i\neq j$, the range of the operators $\mathsf{U}_{k,i}$ and $\mathsf{U}_{k,j}$ are orthogonal. Then, for every $\mathsf{x}$, $\mathsf{v}\in\mathsf{H}$,

[TABLE]

Proof.

i: For every $k\in[p]$, let $\mathsf{U}_{k}\colon\mathsf{H}\to\mathsf{G}_{k}$, $\mathsf{U}_{k}\mathsf{x}=\sum_{i=1}^{m}\mathsf{U}_{k,i}\mathsf{x}_{i}$. Let $\mathsf{x},\mathsf{v}\in\mathsf{H}$. We have

[TABLE]

Therefore, we have

[TABLE]

ii: It follows from (A.5) with $\mathsf{v}=\mathsf{J}_{i}\mathsf{v}_{i}$ that

[TABLE]

hence $\tilde{L}_{i}=\lVert\sum_{k=1}^{p}L^{(k)}\mathsf{U}_{k,i}^{\top}\mathsf{U}_{k,i}\rVert$ is a Lipschitz constant of $\nabla_{i}f(\mathsf{x}_{i},\dots,\mathsf{x}_{i-1},\cdot,\mathsf{x}_{i+1},\dots,\mathsf{x}_{m})$.

iii: Since $\langle\mathsf{U}_{k,i}\mathsf{v}_{i},\mathsf{U}_{k,j}\mathsf{v}_{j}\rangle=0$ if $i\neq j$, it follows from (A.4) that

[TABLE]

Remark A.7**.**

If $R(\mathsf{U}_{k,i})$ and $R(\mathsf{U}_{k,j})$ are orthogonal to each other, then

[TABLE]

and hence $\lVert\sum_{k=1}^{p}L^{(k)}\mathsf{U}_{k}^{\top}\mathsf{U}_{k}\rVert=\max_{1\leq i\leq m}\tilde{L}_{i}$.

Appendix B Additional proofs

Proof. of Theorem 3.1

ii: It follows from (A.2) that, point-wise it holds

[TABLE]

Moreover, since \beta_{2}=\operatorname*{\text{\rm ess\>sup}}\big{(}\max_{1\leq k\leq p}\textstyle\sum_{i\in I_{k}}\varepsilon_{i}\big{)}, we have that $L_{i}\max_{1\leq k\leq p}\textstyle\sum_{i\in I_{k}}\varepsilon_{i}\leq L_{i}\beta_{2}\leq\nu_{i}$ $\mathsf{P}$-a.s. The statement follows.

i It follows by taking the expectation in (B.1) and noting that

[TABLE]

where we used the fact that for every discrete random variable $\zeta$, $\mathsf{E}[\varepsilon_{i}\zeta]=\mathsf{E}[\zeta\,|\,\varepsilon_{i}=1]\mathsf{p}_{i}$.

iii It follows from Proposition A.6iii that 1 holds with $\nu_{i}=\tilde{L}_{i}$.

iv: Let $i\in[m]$ and $\mathsf{v}_{i}\in\mathsf{H}_{i}$ and set $\mathsf{v}=\mathsf{J}_{i}\mathsf{v}_{i}=(0,\dots,0,\mathsf{v}_{i},0,\dots,0)$. Then

[TABLE]

Hence, it follows from 1 that ${\mathsf{f}}(\mathsf{x}+\mathsf{J}_{i}\mathsf{v}_{i})\leq{\mathsf{f}}(\mathsf{x})+\langle\nabla_{i}{\mathsf{f}}(\mathsf{x}),\mathsf{v}_{i}\rangle+(1/2)\nu_{i}\lVert\mathsf{v}_{i}\rVert^{2}$. This shows that ${\mathsf{f}}$ is Lipschitz smooth w.r.t. the $i$-th block coordinate with Lipschitz constant $\nu_{i}$. The global Lipschitz smoothness of ${\mathsf{f}}$ follows from Corollary A.2. ∎

Proof. of Remark 3.2iv.

We have ${\mathsf{f}}(\mathsf{x})=\sum_{k=1}^{p}{\mathsf{g}}_{k}(\mathsf{U}_{k}\mathsf{x})$ and {\mathsf{f}}(\mathsf{x}+\mathsf{v})=\sum_{k=1}^{p}{\mathsf{g}}_{k}\big{(}\mathsf{U}_{k}\mathsf{x}+\sum_{i=1}^{m}\mathsf{U}_{k,i}\mathsf{v}_{i}\big{)}. Moreover, $\nabla{\mathsf{f}}(\mathsf{x})=\sum_{k=1}^{p}\mathsf{U}_{k}^{\top}\nabla{\mathsf{g}}_{k}(\mathsf{U}_{k}\mathsf{x})$. We let $\mathsf{x}\in\mathsf{H}$ and define

[TABLE]

We clearly have $\varphi(\mathsf{v})\geq 0$, $\varphi(0)=0$ and $\psi_{k}(\mathsf{u})\geq 0$, $\psi_{k}(0)=0$. Moreover,

[TABLE]

Therefore,

[TABLE]

Now, if $\omega\in\Omega$ is such that $\sum_{i\in I_{k}}\varepsilon_{i}(\omega)=t$ and we set $I=\{i\in[m]\,|\,\varepsilon_{i}(\omega)=1\}$, then we have $\mathrm{card}(I\cap I_{k})=t$ and

[TABLE]

Hence \psi_{k}\big{(}\sum_{i\in I_{k}}\mathsf{U}_{k,i}\varepsilon_{i}\mathsf{v}_{i}\big{)}\leq(1/t)\sum_{i=1}^{m}\varepsilon_{i}\psi_{k}(t\mathsf{U}_{k,i}\mathsf{v}_{i}) on the event $\sum_{i\in I_{k}}\varepsilon_{i}=t$. Then,

[TABLE]

Plugging the above inequality in (B.3) we get

[TABLE]

where in the last inequality we used 1. So, setting $\beta_{1,i}$ as in (3.3), if $\nu_{i}\geq\beta_{1,i}L_{i}$, then 1 holds. Note that in deriving (B), if for some $i\in[m]$ there are no $k\in[p]$ such that $i\in I_{k}$, the corresponding term \max_{k\in\varnothing}\mathsf{P}\big{(}\sum_{i\in I_{k}}\varepsilon_{i}=t\,|\,\varepsilon_{i}=1\big{)} can be set to zero. ∎

Proof. of formula (3.4).

Since in the proof of (3.3) in Remark 3.2iv, we only use the fact that $i\notin I_{k}\ \Rightarrow\ \mathsf{U}_{k,i}=0$, we can assume, without loss of generality, that, for every $k\in[p]$, $\mathrm{card}(I_{k})=\eta$. Let $i\in\bigcup_{k=1}^{p}I_{k}$. Then, since the block sampling is doubly uniform, we have that \mathsf{P}\big{(}\sum_{i\in I_{k}}\varepsilon_{i}=t\,|\,\varepsilon_{i}=1\big{)} does not depend on $k$ such that $i\in I_{k}$. Therefore, (3.3) becomes \beta_{1,i}=\sum_{t=1}^{\eta}t\mathsf{P}\big{(}\sum_{j\in I_{k}}\varepsilon_{j}=t\,|\,\varepsilon_{i}=1\big{)}, for some $k\in[p]$ such that $i\in I_{k}$. Hence

[TABLE]

where $\mathsf{p}=\mathsf{P}(\varepsilon_{i}=1)$ and $\tilde{\mathsf{p}}=\mathsf{P}(\varepsilon_{j}=1,\varepsilon_{i}=1)$ (with $i\neq j$). Now, since $\mathsf{E}[\sum_{i=1}^{m}\varepsilon_{i}]=m\mathsf{p}$ and $\mathsf{E}[(\sum_{i=1}^{m}\varepsilon_{i})^{2}]=\sum_{i=1}^{m}\mathsf{E}[\varepsilon_{i}]+\sum_{i\neq j}\mathsf{E}[\varepsilon_{i}\varepsilon_{j}]=m\mathsf{p}+m(m-1)\tilde{\mathsf{p}}$, we have that \tilde{\mathsf{p}}=\big{(}\mathsf{E}[(\sum_{i=1}^{m}\varepsilon_{i})^{2}]-m\mathsf{p}\big{)}/(m(m-1)), which plugged into (B.6) gives (3.4).

Proof. of Lemma 4.3

Let $\mathsf{z}\in\mathsf{H}$. It follows from the definition of $\mathsf{x}^{+}$ that $\mathsf{x}-\mathsf{x}^{+}-\nabla\varphi(\mathsf{x})\in\partial\psi(\mathsf{x}^{+})$. Therefore, $\psi(\mathsf{z})\geq\psi(\mathsf{x}^{+})+\langle\mathsf{x}-\mathsf{x}^{+}-\nabla\varphi(\mathsf{x}),\mathsf{z}-\mathsf{x}^{+}\rangle+(\mu_{\psi}/2)\lVert\mathsf{z}-\mathsf{x}^{+}\rVert^{2},$ hence

[TABLE]

Now, we note that $\lVert\mathsf{x}^{+}-\mathsf{z}\rVert^{2}=\lVert\mathsf{x}^{+}-\mathsf{x}\rVert^{2}+\lVert\mathsf{x}-\mathsf{z}\rVert^{2}+2\langle\mathsf{x}^{+}-\mathsf{x},\mathsf{x}-\mathsf{z}\rangle$. Then,

[TABLE]

and hence

[TABLE]

Since $\langle\nabla\varphi(\mathsf{x}),\mathsf{z}-\mathsf{x}\rangle\leq\varphi(\mathsf{z})-\varphi(\mathsf{x})-(\mu_{\varphi}/2)\lVert\mathsf{z}-\mathsf{x}\rVert^{2}$, the statement follows. ∎

Lemma B.1**.**

Let $a,b,c\in\mathbb{R}_{++}$. Then the largest constant $\bar{\lambda}>0$ satisfying the following inequality

[TABLE]

is

[TABLE]

Proof.

Property (B.7) is equivalent to

[TABLE]

Therefore,

[TABLE]

Now, since

[TABLE]

it follows from (B.9) that

[TABLE]

Therefore, the statement follows. ∎

Proof. of Theorem 4.10

We first note that, since, $\lVert\cdot\rVert^{2}_{{\mathsf{\Gamma}}^{-1}}\geq\mathsf{p}_{\min}\lVert\cdot\rVert^{2}_{\mathsf{W}}$, the conclusion of Proposition 4.6iii can be stated as follows:

[TABLE]

Let $\mathsf{x}=\mathsf{x}_{*}$ and set for brevity $r_{n}^{2}=(\mathsf{p}_{\min}/2)\lVert x^{n}-\mathsf{x}_{*}\rVert_{\mathsf{W}}^{2}$, and $F_{n}={\mathsf{F}}(x^{n})$. Then, (B.11) yields

[TABLE]

Let $b=1+(\delta-1)_{+}/(2+\sigma_{\mathsf{\Gamma}^{-1}}-\delta)$. Then the above inequality can be rewritten as

[TABLE]

Now, we derive from (2.1)-(2.2) that

[TABLE]

Therefore, it follows from Lemma B.1 (with $c=\mu_{\mathsf{\Gamma}^{-1}}+\sigma_{\mathsf{\Gamma}^{-1}}$ and $a=1+\sigma_{\mathsf{\Gamma}^{-1}}$) that

[TABLE]

where

[TABLE]

Then, by (B.12) and (B.13), we have that

[TABLE]

and hence, taking the expectation, and applying the resulting inequality recursively, we have,

[TABLE]

To conclude it is sufficient to note that, since

[TABLE]

we have

[TABLE]

Appendix C Some results on duality theory

In this section, for the reader’s convenience, we recap the results obtained in [9]. Let $\varphi\colon\mathsf{H}\to\mathbb{R}$ and $\psi\colon\mathsf{G}\to\left]-\infty,+\infty\right]$ be two lower semicontinuous and convex functions defined on Hilbert spaces, and let $\mathsf{A}\colon\mathsf{H}\to\mathsf{G}$ be a bounded linear operator. In this section we suppose that $\varphi$ is $\mu$-strongly convex. We consider the following optimization problems in duality (in the sense of Fenchel-Rockafellar)

[TABLE]

We define the duality gap function $G\colon\mathsf{H}\times\mathsf{G}\to\left]-\infty,+\infty\right]$, $G(\mathsf{x},\mathsf{u})=\mathcal{P}(\mathsf{x})+\mathcal{D}(\mathsf{u})$ and recall that

[TABLE]

So, the duality gap function bounds the primal and dual objectives. We have the following theorem

Theorem C.1**.**

Suppose that $R(\mathsf{A})\subset\operatorname*{\text{\rm dom}}\partial\psi$. Then the following holds:

1. (i)

Suppose that $\psi^{*}$ is $\alpha$-strongly convex. Let $\mathsf{u}\in\operatorname*{\text{\rm dom}}\psi^{*}$ and set $\mathsf{x}=\nabla\varphi^{*}(-\mathsf{A}^{\top}\mathsf{u})$. Then,

[TABLE] 2. (ii)

Suppose that $\psi$ is $\theta$-Lipschitz continuous. Let $\mathsf{u}\in\operatorname*{\text{\rm dom}}\psi^{*}$ be such that $\mathcal{D}(\mathsf{u})-\inf\mathcal{D}<\lVert\mathsf{A}\rVert^{2}L^{2}/\mu$ and set $\mathsf{x}=\nabla\varphi^{*}(-\mathsf{A}^{\top}\mathsf{u})$. Then, we have

[TABLE]

Moreover, if $u$ is a random variable taking values in $\operatorname*{\text{\rm dom}}\psi^{*}$ and such that $\mathsf{E}[\mathcal{D}(u)]-\inf\mathcal{D}<\lVert\mathsf{A}\rVert^{2}L^{2}/\mu$ and we set $x=\nabla\varphi^{*}(-\mathsf{A}^{\top}u)$, then $\mathsf{E}[G(x,u)]\leq 2\lVert\mathsf{A}\rVert\theta/\mu^{1/2}(\mathsf{E}[\mathcal{D}(u)]-\inf\mathcal{D})^{1/2}$.