Parallel algorithms and probability of large deviation for stochastic optimization problems

TL;DR

This paper investigates parallel stochastic optimization algorithms under limited function value information, demonstrating that parallelization with averaging maintains low deviation probability and reduces computational time.

Contribution

It introduces bounds for large deviation probabilities in parallel stochastic optimization without access to objective function values, extending previous results to more practical scenarios.

Findings

Parallelization with averaging achieves similar deviation bounds as serial methods.

Under light-tail assumptions, parallel algorithms reduce computational time.

Results apply even when objective function values are unavailable or costly to compute.

Abstract

We consider convex stochastic optimization problems under different assumptions on the properties of available stochastic subgradient. It is known that, if the value of the objective function is available, one can obtain, in parallel, several independent approximate solutions in terms of the objective residual expectation. Then, choosing the solution with the minimum function value, one can control the probability of large deviation of the objective residual. On the contrary, in this short paper, we address the situation, when the value of the objective function is unavailable or is too expensive to calculate. Under "`light-tail"' assumption for stochastic subgradient and in general case with moderate large deviation probability, we show that parallelization combined with averaging gives bounds for probability of large deviation similar to a serial method. Thus, in these cases, one can benefit from parallel computations and reduce the computational time without loss in the solution quality.

Equations

$$\underset{x\in Q\subset E}{\min }\left\{f\left(x\right):={\mathbb{E}}_{\xi }\left[f\left(x,\xi \right)\right]\right\},$$

$${\mathbb{E}}_{\xi }\left[{\nabla }_{x}f\left(x,\xi \right)\right]\in \partial f\left(x\right),{\mathbb{E}}_{\xi }\left[{\Vert {\nabla }_{x}f\left(x,\xi \right)\Vert }_{\ast }^2\right]\le M^2$$

$$x^{k+1}={\mathrm{Mirr}}_{x^k}\left(h{\nabla }_{x}f\left(x^k,{\xi }^k\right)\right),{\mathrm{Mirr}}_{x^k}\left(v\right):=\arg \underset{x\in Q}{\min }\left\{⟨v,x-x^k⟩+V_{x^k}\left(x\right)\right\},$$

$$2V_{x^{k+1}}\left(x\right)\le 2V_{x^k}\left(x\right)+2h⟨{\nabla }_{x}f\left(x^k,{\xi }^k\right),x-x^k⟩+h^2{\Vert {\nabla }_{x}f\left(x^k,{\xi }^k\right)\Vert }_{\ast }^2,\forall x\in Q.$$

$$f\left(x^k\right)-f\left(x\right)\le ⟨\nabla f(x^k),x^k-x⟩\le ⟨\nabla f(x^k)-{\nabla }_{x}f\left(x^k,{\xi }^k\right),x^k-x⟩+$$

$$+\frac{1}{h}\left(V_{x^k}\left(x\right)-V_{x^{k+1}}\left(x\right)\right)+\frac{h}{2}{\Vert {\nabla }_{x}f\left(x^k,{\xi }^k\right)\Vert }_{\ast }^2.$$

$$f\left(x^k\right)-f\left(x\right)\le \frac{1}{h}\left(V_{x^k}\left(x\right)-\mathbb{E}\left[V_{x^{k+1}}\left(x\right)|{\xi }^1,...,{\xi }^{k-1}\right]\right)+\frac{h}{2}\underset{\stackrel{\text{eqref}eq:As}{\le }{M}^2}{\underbrace{\mathbb{E}\left[{\Vert {\nabla }_{x}f\left(x^k,{\xi }^k\right)\Vert }_{\ast }^2|{\xi }^1,...,{\xi }^{k-1}\right]}}.$$

$$\mathbb{E}\left[f\left({\bar{x}}^N\right)\right]-f_{\ast }\le \mathbb{E}\frac{1}{N}\sum _{k=0}^{N-1}f(x^k)-f_{\ast }\le \frac{1}{hN}{V}_{x^0}\left(x_{\ast }\right)+\frac{M^{2}h}{2}\le \sqrt{\frac{2M^2{R}^2}{N}},$$

$$R\text{is s.t.}{V}_{x^0}\left(x_{\ast }\right)\le R^2,{\bar{x}}^N:=\frac{1}{N}\sum _{k=0}^{N-1}{x}^k,h=\frac{R}{M}\sqrt{\frac{2}{N}}.$$

$$N=\lceil \frac{2M^2{R}^2}{{\epsilon }^2}\rceil ,$$

$$\mathbb{P}\left\{f\left({\bar{x}}^N\right)-f_{\ast }\le \frac{C_{1}M}{\sqrt{N}}\left(R+C_2\bar{R}\sqrt{\ln \left(1/\sigma \right)}\right)\right\}\ge 1-\sigma ,$$

$$\mathbb{P}\left\{f\left({\bar{x}}^N\right)-f_{\ast }\ge c\right)\le \mathbb{P}\left\{\eta \ge c\right\},$$

$$\mathbb{P}\left(f\left({\bar{x}}^N\right)-f_{\ast }\ge \epsilon \right)\le \frac{\mathbb{E}\left[f\left({\bar{x}}^N\right)\right]-f_{\ast }}{\epsilon }\le \frac{1}{2}.$$

$$\mathbb{P}\left\{f\left({\bar{x}}^{N,i}\right)-f_{\ast }\ge \epsilon \right\}\le \mathbb{P}\left\{{\eta }_i\ge \epsilon \right\},{\eta }_i\in \mathrm{N}\left(\frac{\epsilon }{2},\frac{{\epsilon }^2}{4}\right).$$

$$\mathbb{P}\left\{f\left({\bar{x}}^K\right)-f_{\ast }\ge \epsilon \right\}\le \mathbb{P}\left\{\frac{1}{K}\sum _{k=i}^K\left(f\left({\bar{x}}^{N,i}\right)-f_{\ast }\right)\ge \epsilon \right\}\stackrel{\text{eqref}eq:Th{1}_{p}r}{\le }$$

$$\mathbb{P}\left\{\frac{1}{K}\sum _{i=1}^K{\eta }_i\ge \frac{\epsilon }{2}+\frac{\epsilon }{2}\right)=\mathbb{P}\left\{\left(\frac{1}{K}\sum _{i=1}^K{\eta }_i\right)-\frac{\epsilon }{2}\ge \frac{\epsilon }{2}\right\}=\mathbb{P}\left\{\eta -\frac{\epsilon }{2}\ge \frac{\epsilon }{2}\sqrt{K}\right\}\le \sigma ,$$

$$\mathrm{N}\left(\frac{\epsilon }{2},\frac{{\epsilon }^2}{4}\right)+...+\mathrm{N}\left(\frac{\epsilon }{2},\frac{{\epsilon }^2}{4}\right)\stackrel{d}{=}\mathrm{N}\left(\frac{K\epsilon }{2},\frac{K{\epsilon }^2}{4}\right),\frac{1}{K}\mathrm{N}\left(\frac{K\epsilon }{2},\frac{K{\epsilon }^2}{4}\right)\stackrel{d}{=}\mathrm{N}\left(\frac{\epsilon }{2},\frac{{\epsilon }^2}{4K}\right),$$

Full text

Keywords Stochastic Convex Optimization, Probability of Large Deviation, Mirror Descent, Parallel Algorithm

Mathematics Subject Classification 90C15, 90C25

1 Introduction

We consider the following general stochastic optimization problem over a convex compact set $Q$:

[TABLE]

where $E$ is a finite-dimensional real vector space, $\xi$ is a random vector, $f\left({x,\xi}\right)$ is a closed convex function w.r.t $x$ for a.e. $\xi$, and $x$ and $\xi$ are independent. Under these assumptions, this problem is a convex optimization problem.

Our main goal is to approximately solve this problem using some algorithm. Usually, in the stochastic optimization literature [1, 2], two measures for the quality of an approximate solution $\bar{x}$ are considered. The first is expectation of the objective residual. In this case, $\bar{x}$ is an $\varepsilon$-solution of (1) for $\varepsilon>0$ iff ${\mathbb{E}}f(\bar{x})-f_{*}\leq\varepsilon$, where $f_{*}$ is the optimal value in (1) and expectation is taken with respect to all the randomness arising in the algorithmic process. The second is bound for probability of large deviation of the objective residual. In this case $\bar{x}$, is an $(\varepsilon,\sigma)$-solution of (1) for $\varepsilon>0$, $\sigma\in(0,1)$ iff ${\mathbb{P}}\{f(\bar{x})-f_{*}>\varepsilon\}\leq\sigma$. We mainly focus on the latter quality measure in this paper.

It is known that, if the value of the objective function is available (e.g. in randomized methods [3]), one can obtain in parallel logarithmic in $\sigma^{-1}$ number of independent $\varepsilon$-solutions. Then, the solution with the minimum function value is an $(\varepsilon,\sigma)$-solution. Nevertheless, the value of the objective function can be unavailable or too expensive to calculate or approximate. The latter is easy to imagine since, for $\xi\in{\mathbb{R}}^{p}$, the computational effort for calculation of $\bar{f}$ s.t. $|\bar{f}-f(x)|\leq\delta$ can amount up to $O\left(\delta^{-p}\right)$ calculations of $f(x,\xi)$ at different $\xi$. Our goal is to propose a technique, which allows to obtain an $(\varepsilon,\sigma)$-solution based on a number computed in parallel $\varepsilon$-solutions without calculation of the function $f(x)$ value.

Our approach is based on Stochastic Mirror Descent algorithm [2]. It turns out that, under some mild assumptions, an $\varepsilon$-solution to (1), obtained by Stochastic Mirror Descent, is also an $(\tilde{\varepsilon},\sigma)$-solution. We use this fact and calculate in parallel logarithmic in $\sigma^{-1}$ number of independent $\varepsilon$-solutions, average them and prove that this average is an $(\varepsilon,\sigma)$-solution. Thus, we can benefit from parallelization and reduce the computation time without any loss in the solution quality.

2 Stochastic Mirror Descent

This section is devoted to description of Stochastic Mirror Descent (SMD) [2, 3] and its convergence properties in terms of expectation of the objective residual and also in terms of probability of large deviation of this residual. These convergence results provide the basis of our approach for constructing an $(\varepsilon,\sigma)$-solution by parallelization. Let us choose some norm $\left\|\cdot\right\|$ on $E$ and denote the conjugate norm by $\left\|\cdot\right\|_{\ast}$. We assume that, at any point $x\in Q$, a stochastic subgradient $\nabla_{x}f\left({x,\xi}\right)$ of $f(x)$ is available and satisfies

[TABLE]

for some constant $M>0$. We choose a prox-function $d\left(x\right)$ which is 1-strongly convex in $\left\|\cdot\right\|$. Let $x^{0}=\arg\min_{x\in Q}d(x)$. W.l.o.g. we assume that $d(x^{0})=0$. The algorithm uses Bregman’s divergence $V_{z}\left({x}\right)=d\left(x\right)-d\left(z\right)-\left\langle{\nabla d\left(z\right),x-z}\right\rangle$. Let $x_{*}$ be a solution of (1), $R$ be a number s.t. $V_{x^{0}}\left({x_{\ast}}\right)\leq R^{2}$, and $\bar{R}$ be a number s.t. $\max_{x\in Q}V_{x}\left({x_{\ast}}\right)\leq\bar{R}$.

Stochastic Mirror Descent [2, 3] iterates as follows, starting from $x^{0}\in Q$,

[TABLE]

where $h>0$ is the stepsize, $\left\{{\xi^{k}}\right\}_{k\geq 0}$ is an i.i.d. sample of $\xi$. The main property of SMD-step [3] is

[TABLE]

Further, using convexity of $f(x)$, for any $\nabla f(x^{k})\in\partial f(x^{k})$ and $x\in Q$,

[TABLE]

[TABLE]

Taking conditional expectation w.r.t. $\xi^{1},...,\xi^{k-1}$ and using (2), one obtains

[TABLE]

Since $\left\{{\xi^{k}}\right\}_{k\geq 0}$ is an i.i.d. sample, taking full expectation from the both sides of these inequalities for $k=0,...,N-1$, summing them up and taking $x=x_{\ast}$ gives, by convexity of $f\left(x\right)$,

[TABLE]

where

[TABLE]

Choosing

[TABLE]

we obtain that $\bar{x}^{N}$ satisfies ${\mathbb{E}}\left[{f\left({\bar{x}^{N}}\right)}\right]-f_{\ast}\leq\varepsilon$ and, hence, is an $\varepsilon$-solution. Note that this bound for $N$ is optimal [3] up to a multiplicative constant factor for the class of convex stochastic programming problems (1) with a.e. bounded stochastic subgradients.

It turns out that it is possible to prove bounds for probability of large deviation for $f\left({\bar{x}^{N}}\right)-f_{\ast}$.

Proposition 1** ([2, 5, 6]).**

Assume that one of the following assumptions holds. a) $\left\|{\nabla_{x}f\left({x,\xi}\right)}\right\|_{\ast}\leq M$ for a.e. $\xi$; b) ${\mathbb{E}}_{\xi}\left({\exp\left({{\left\|{\nabla_{x}f\left({x,\xi}\right)}\right\|_{\ast}^{2}}\mathord{\left/{\vphantom{{\left\|{\nabla_{x}f\left({x,\xi}\right)}\right\|_{\ast}^{2}}{M^{2}}}}\right.\kern-1.2pt}{M^{2}}}\right)}\right)\leq\exp\left(1\right)$ and $\ln\sigma^{-1}\ll N$; c) There exists some $\alpha>2$ s.t., for all, $t\geq 0$, ${\mathbb{P}}\left({\frac{\left\|{\nabla f\left({x,\xi}\right)}\right\|_{2}^{2}}{M^{2}}\geq t}\right)\leq\frac{1}{\left({t+1}\right)^{\alpha}}$, and $\sigma^{-1/\left({\alpha-1}\right)}\ll N$. Then the point $\bar{x}^{N}$ generated by SMD (3), (4) satisfies

[TABLE]

where in the case a) $C_{1}=\sqrt{2}$, $C_{2}=2\sqrt{2}$; in the case b) $C_{1}=C_{2}=2\sqrt{2}$; in the case c) $C_{1}=C_{1}(\alpha)$, $C_{2}=1$.

Corollary 1**.**

Let any of three assumptions of Proposition 1 hold. Choose $N=\left\lceil\frac{CM^{2}\bar{R}^{2}}{\varepsilon^{2}}\right\rceil$, where the constant $C$ depends on $C_{1},C_{2}$. Then the point $\bar{x}^{N}$ generated by SMD (3), (4) satisfies, for any $c\geq 0$,

[TABLE]

where $\eta\in{\rm N}\left({\varepsilon,\varepsilon^{2}}\right)$ – normal random variable with mean $\varepsilon$ and variance $\varepsilon^{2}$.

3 Parallelization and bounds for probability of large deviation

In this section, we first discuss a known way to obtain an $(\varepsilon,\sigma)$-solution using a number of $\varepsilon$-solutions calculated in parallel. Then, we suggest a new way of doing this without calculation of the objective $f(x)$ value, state and prove the main result.

Assume that $\bar{x}^{N}$ is an $\varepsilon/2$-solution, obtained by SMD (3), (4) with $N=\left\lceil\frac{8M^{2}R^{2}}{\varepsilon^{2}}\right\rceil$. Then, using the Markov inequality [5], we obtain

[TABLE]

If one calculates in parallel $K=\left\lceil\log_{2}\left({\sigma^{-1}}\right)\right\rceil$ independent SMD $\varepsilon/2$-solutions $\left\{{\bar{x}^{N,i}}\right\}_{i=1}^{K}$ and chooses the one $\bar{x}_{\min}^{N}$ which minimizes $f\left({\bar{x}^{N,i}}\right)$, then, in total $\left\lceil\frac{8M^{2}R^{2}}{\varepsilon^{2}}\right\rceil\left\lceil\log_{2}\left({\sigma^{-1}}\right)\right\rceil$ calculations of stochastic subgradient, one obtains ${\mathbb{P}}\left({f\left({\bar{x}_{\min}^{N}}\right)-f_{\ast}\geq\varepsilon}\right)\leq\sigma$. Thus, $\bar{x}_{\min}^{N}$ is an $(\varepsilon,\sigma)$-solution of (1). The crucial point here is the possibility to calculate the value of the function $f\left(x\right)$.

We now suggest a technique which does not rely on the assumption of the function $f\left(x\right)$ value availability. This assumption may not hold [1] in many real stochastic programming problems, e.g. in maximum likelihood approach used in mathematical statistics.

Theorem 1**.**

Let any of three assumptions of Proposition 1 hold. Let $K=\left\lceil 2\ln\left({\sigma^{-1}}\right)\right\rceil$ and $\left\{{\bar{x}^{N,i}}\right\}_{i=1}^{K}$ be independent points obtained by SMD (3), (4) with $N=\left\lceil\frac{4CM^{2}\bar{R}^{2}}{\varepsilon^{2}}\right\rceil$. Then the point $\bar{x}^{K}=\frac{1}{K}\sum\limits_{i=1}^{K}{\bar{x}^{N,i}}$ is an $(\varepsilon,\sigma)$-solution of (1).

Proof Using Corollary 1 with $c=\varepsilon$, we obtain, for all $i=1,...,K$,

[TABLE]

By convexity of $f(x)$, since $\left\{{\bar{x}^{N,i}}\right\}_{i=1}^{K}$ are i.i.d., $\left\{{\eta_{i}}\right\}_{i=1}^{K}$ are i.i.d.,

[TABLE]

where $\eta\in{\rm N}\left({\frac{\varepsilon}{2},\frac{\varepsilon^{2}}{4}}\right)$. Here we used well-known facts about the properties of sum of independent normal random variables [4]

[TABLE]

that $K=\left\lceil 2\ln\left({\sigma^{-1}}\right)\right\rceil$ and, for $\eta\in{\rm N}\left({\frac{\varepsilon}{2},\frac{\varepsilon^{2}}{4}}\right)$, ${\mathbb{P}}\left\{{\eta-\frac{\varepsilon}{2}\geq\frac{\varepsilon}{2}\sqrt{2\ln\left({\sigma^{-1}}\right)}}\right\}\leq\sigma$. Thus, the point $\bar{x}^{K}$ is an $(\varepsilon,\sigma)$-solution of (1). ∎

To sum up, under some mild assumptions, but without calculation of the objective function value, we propose a way to obtain an $(\varepsilon,\sigma)$-solution using a number of $\varepsilon$-solutions calculated in parallel. This approach allows to reduce the computational time without any loss in the solution quality. At the same time, we answer to the question of Yu. Nesterov [7]. The question can be stated as follows. When the quality of a solution of problem (1) obtained by one wise old man, thinking for $\Theta\left({M^{2}\bar{R}^{2}\ln\left({\sigma^{-1}}\right)}/\varepsilon^{2}\right)$ days, is the same as obtained by $\Theta\left(\ln\left({\sigma^{-1}}\right)\right)$ experts, each thinking for $\Theta\left(M^{2}\bar{R}^{2}/\varepsilon^{2}\right)$ days? Our answer is that the quality is equivalent under any of three assumptions of Proposition 1.