Surrogate Losses for Online Learning of Stepsizes in Stochastic Non-Convex Optimization

TL;DR

This paper introduces surrogate loss functions that enable online learning of adaptive stepsizes for stochastic gradient descent in non-convex optimization, improving convergence without manual tuning.

Contribution

It proposes a novel surrogate loss framework that casts stepsize selection as an online convex optimization problem, allowing automatic, theoretically-guaranteed adaptation.

Findings

Achieves convergence rates adaptive to noise levels.

Provides a self-tuning SGD algorithm with theoretical guarantees.

Reduces the need for manual stepsize tuning in practice.

Abstract

Stochastic Gradient Descent (SGD) has played a central role in machine learning. However, it requires a carefully hand-picked stepsize for fast convergence, which is notoriously tedious and time-consuming to tune. Over the last several years, a plethora of adaptive gradient-based algorithms have emerged to ameliorate this problem. They have proved efficient in reducing the labor of tuning in practice, but many of them lack theoretic guarantees even in the convex setting. In this paper, we propose new surrogate losses to cast the problem of learning the optimal stepsizes for the stochastic optimization of a non-convex smooth objective function onto an online convex optimization problem. This allows the use of no-regret online algorithms to compute optimal stepsizes on the fly. In turn, this results in a SGD algorithm with self-tuned stepsizes that guarantees convergence rates that are automatically adaptive to the level of noise.

Figures

Equations

$${\mathbf{x}}_{t+1}={\mathbf{x}}_t-{\eta }_t\mathbf{g}({\mathbf{x}}_t,{\xi }_t),$$

$${\mathbb{E}}_t\left[\mathbf{g}({\mathbf{x}}_t,{\xi }_t)\right]={\mathbb{E}}_t\left[\mathbf{g}({\mathbf{x}}_t,{\xi }_t^{\prime })\right]=\nabla f({\mathbf{x}}_t),$$

$${\mathbb{E}}_t\left[⟨\mathbf{g}({\mathbf{x}}_t,{\xi }_t),\mathbf{g}({\mathbf{x}}_t,{\xi }_t^{\prime })⟩\right]=\Vert \nabla f({\mathbf{x}}_t){\Vert }^2.$$

$${\mathbb{E}}_t\left[{\Vert \mathbf{g}({\mathbf{x}}_t,{\xi }_t)-\nabla f({\mathbf{x}}_t)\Vert }^2\right]$$

$$\left|f({\mathbf{x}}_2)-f({\mathbf{x}}_1)-⟨\nabla f({\mathbf{x}}_1),{\mathbf{x}}_2-{\mathbf{x}}_1⟩\right|\le \frac{M}{2}\Vert {\mathbf{x}}_2-{\mathbf{x}}_1{\Vert }^2.$$

$$\Vert \nabla f(\mathbf{x}){\Vert }^2\ge 2\mu (f(\mathbf{x})-f^{\star }),\forall \mathbf{x}.$$

$$\mathbb{E}[\Vert \nabla f({\mathbf{x}}_k){\Vert }^2]\le O\left((f({\mathbf{x}}_1)-f^{\star })/(\eta T)+\eta {\sigma }^2\right),$$

$${\ell }_t(\eta )=\frac{M{\eta }^2}{2}\Vert \mathbf{g}({\mathbf{x}}_t,{\xi }_t){\Vert }^2-\eta ⟨\mathbf{g}({\mathbf{x}}_t,{\xi }_t),\mathbf{g}({\mathbf{x}}_t,{\xi }_t^{\prime })⟩,$$

$$\mathbb{E}\left[f({\mathbf{x}}_{t+1})-f({\mathbf{x}}_t)\right]\le \mathbb{E}\left[{\ell }_t({\eta }_t)\right].$$

$$\mathbb{E}\left[f({\mathbf{x}}_{t+1})-f({\mathbf{x}}_t)\right]$$

$$\le \mathbb{E}\left[⟨\nabla f({\mathbf{x}}_t),{\mathbf{x}}_{t+1}-{\mathbf{x}}_t⟩+\frac{M}{2}\Vert {\mathbf{x}}_{t+1}-{\mathbf{x}}_t{\Vert }^2\right]$$

$$=\mathbb{E}\left[⟨\nabla f({\mathbf{x}}_t),-{\eta }_t\mathbf{g}({\mathbf{x}}_t,{\xi }_t)⟩+\frac{M}{2}{\eta }_t^2\Vert \mathbf{g}({\mathbf{x}}_t,{\xi }_t){\Vert }^2\right]$$

$$=\mathbb{E}\left[⟨\nabla f({\mathbf{x}}_t),{\mathbb{E}}_t\left[-{\eta }_t\mathbf{g}({\mathbf{x}}_t,{\xi }_t)\right]⟩+\frac{M}{2}{\eta }_t^2\Vert \mathbf{g}({\mathbf{x}}_t,{\xi }_t){\Vert }^2\right].$$

$$\mathbb{E}$$

$$=\mathbb{E}\left[{\mathbb{E}}_t\left[⟨\mathbf{g}({\mathbf{x}}_t,{\xi }_t^{\prime }),-{\eta }_t\mathbf{g}({\mathbf{x}}_t,{\xi }_t)⟩\right]\right]$$

$$=\mathbb{E}\left[⟨\mathbf{g}({\mathbf{x}}_t,{\xi }_t^{\prime }),-{\eta }_t\mathbf{g}({\mathbf{x}}_t,{\xi }_t)⟩\right].$$

$${\mathrm{Regret}}_T(\eta )=\sum _{t=1}^T({\ell }_t({\eta }_t)-{\ell }_t(\eta )).$$

$$\mathbb{E}$$

$$\le f({\mathbf{x}}_1)-f^{\star }+\mathbb{E}\left[{\mathrm{Regret}}_T(\eta )\right]+\frac{M{\eta }^2}{2}\sum _{t=1}^T\mathbb{E}[{\sigma }_t^2].$$

$$f^{\star }-f({\mathbf{x}}_1)\le$$

$$=$$

$$\le$$

$$=$$

$$\le$$

$${\mathbb{E}}_t[{\ell }_t(\eta )]$$

$${\mathbf{v}}_{t+1}=\underset{\mathbf{v}\in {\mathbb{R}}^d}{\mathrm{argmin}}r(\mathbf{v})+\sum _{s=1}^t{\ell }_s(\mathbf{v}),$$

$${\mathrm{Regret}}_T({\mathbf{x}}^{\star })\le r({\mathbf{x}}^{\star })+\frac{1}{2}\sum _{t=1}^T\Vert {\mathbf{g}}_t{\Vert }_{(t),\star }^2,$$

$$\mathbb{E}$$

$$\le \frac{2M}{T}\left(f({\mathbf{x}}_1)-f^{\star }+\frac{5L^2}{M}\ln \left(1+\frac{L^{2}T}{\alpha }\right)\right)$$

$$+\frac{1}{T}\sqrt{2M\sum _{t=1}^T\mathbb{E}[{\sigma }_t^2]\left(f({\mathbf{x}}_1)-f^{\star }+\frac{\alpha }{2M}\right)}$$

$$+\frac{1}{T}\sqrt{10L^2\sum _{t=1}^T\mathbb{E}[{\sigma }_t^2]\ln \left(1+\frac{L^{2}T}{\alpha }\right)}.$$

$${\eta }_t=\max \left\{0,\min \left\{\frac{\alpha +{\sum }_{j=1}^{t-1}⟨{\mathbf{g}}_j,{\mathbf{g}}_j^{\prime }⟩}{M\left(\alpha +{\sum }_{j=1}^{t-1}\Vert {\mathbf{g}}_j{\Vert }^2\right)},\frac{2}{M}\right\}\right\}.$$

$$\sum _{t=1}^T{a}_{t}h\left(a_0+\sum _{i=1}^t{a}_i\right)$$

$$a_{t}h(s_t)={\int }_{s_{t-1}}^{s_t}h(s_t)dx\le {\int }_{s_{t-1}}^{s_t}h(x)dx.$$

$${\mathrm{Regret}}_T(\eta )$$

$$\le \frac{M\alpha }{2}{\left(\eta -\frac{1}{M}\right)}^2+\frac{1}{2M}\sum _{t=1}^T\frac{{\left({\ell }_t^{\prime }({\eta }_t)\right)}^2}{\alpha +{\sum }_{s=1}^t\Vert {\mathbf{g}}_s{\Vert }^2}.$$

$${\left({\ell }_t^{\prime }({\eta }_t)\right)}^2$$

$$\le 2⟨{\mathbf{g}}_t,{\mathbf{g}}_t^{\prime }{⟩}^2+2M^2{\eta }_t^2\Vert {\mathbf{g}}_t{\Vert }^4$$

Full text

Online Learning, Stochastic Optimization, Non-convex

1 Introduction

In recent years, Stochastic Gradient Descent (SGD) has become the tool of choice for fast optimization of convex and non-convex objective functions. Its simplicity of implementation allows for use in virtually any machine learning problem. In its basic version, it iteratively updates the solution to an optimization problem, moving in the negative direction of the gradient of the objective function at the current solution:

[TABLE]

where $\boldsymbol{g}(\boldsymbol{x}_{t},\xi_{t})$ is a stochastic gradient of the objective function $f$ at the current point $\boldsymbol{x}_{t}$ depending on the stochastic variable $\xi_{t}$. A critical component of the algorithm is the stepsize $\eta_{t}>0$. In order to achieve a fast convergence, the stepsizes must be carefully selected, taking into account the objective function and characteristics of the noise. This task becomes particularly daunting because the noise might change over time due to a variety of factors such as, e.g., approaching the local minimum of the function, changing size of the minibatch, gradients calculated through a simulation.

For the above reasons, a number of variants of SGD have been proposed trying to “adapt” the stepsizes in more or less theoretically principled ways. Indeed, the idea of adapting stepsizes is an old one. A few famous examples are the Polyak’s rule (Polyak, 1987), Stochastic Meta-Descent (Schraudolph, 1999), AdaGrad (Duchi et al., 2011). However, most of previous approaches to adapting the stepsizes are designed for convex functions or without a guaranteed strategy of converging to some optimal stepsize. In fact, often the definition itself of “optimal” stepsize is missing.

In this paper, we take a different and novel route. We study theoretically the setting of stochastic smooth non-convex optimization and we design convex surrogate loss functions that upper bound the expected decrement of the objective function after an SGD update. The first advantage of our approach is that the optimal stepsize can be now defined as the one minimizing the surrogate losses. Moreover, using a no-regret online learning algorithm (Cesa-Bianchi & Lugosi, 2006), we can adapt the stepsizes and guarantee that they will be close to the one of the a-posteriori optimal stepsize. Moreover, basing our approach on online learning methods, we gain the implicit robustness of these methods to adversarial conditions.

The rest of the paper is organized as follows. We begin by discussing related work (Section 2), and then introduce necessary definitions and assumptions (Section 3). Next, we introduce the surrogate loss functions (Section 4) and use them to design an algorithm that adapts global and coordinate-wise stepsizes (Sections 5 and 6). We also empirically validate our theoretical findings on a classification task (Section 7 and Appendix) showing that, differently from other adaptive methods, our method does not require fiddling with stepsizes to guarantee convergence in the stochastic setting. Finally, we draw some conclusions and describe the future work in Section 8.

2 Related Work

Here we discuss the theoretical related work on adaptive stochastic optimization algorithms. First, the convergence of a random iterate of SGD for non-convex smooth functions has been proved by Ghadimi & Lan (2013). They also calculate how the optimal stepsize depends on the variance of the noise in the gradients and the smoothness of the objective function.

The optimal convergence rate was also obtained by Ward et al. (2018) using AdaGrad global stepsizes, without the need to tune parameters. Li & Orabona (2019) improves over their results by removing the assumption of bounded gradients. However, both analyses focus on the adaptivity of non-per-coordinate updates, and are somewhat complicated in order to deal with unbounded gradients or non-independence of the current stepsize from the current step gradient. In comparison, our technique is relatively simple, allowing us to easily show a nontrivial guarantee for per-coordinate updates. In addition, their results cannot recover linear rates of convergence assuming, for example, the Polyak-Łojasiewicz condition (Karimi et al., 2016).

The idea of tuning stepsizes with online learning has been explored in the online convex optimization literature (Koolen et al., 2014; van Erven & Koolen, 2016). There, the possible stepsizes are discretized and an expert algorithm is used to select the stepsize to use online. Instead, in our work the use of convex surrogate loss functions allows us to directly learn the optimal stepsize, without needing to discretize the range of stepsizes. This becomes very important when we consider the possibility of learning a stepsize for each coordinate (Section 6), as we avoid a computational overhead exponential in the dimension of the space $d$ that a discretization would incur.

3 Definitions and Assumptions

We use bold lower-case letters to denote vectors, and bold upper-case letters for matrices, e.g., $\boldsymbol{u}\in\mathbb{R}^{d},\boldsymbol{A}\in\mathbb{R}^{m\times n}$. The ith coordinate of a vector $\boldsymbol{u}$ is $u_{i}$. Throughout this paper, we study the Euclidean space $\mathbb{R}^{d}$ with the inner product $\langle\cdot,\cdot\rangle$. Unless explicitly noted, all norms are the Euclidean norm. The dual norm $\|\cdot\|_{*}$ is the norm defined by $\|\boldsymbol{v}\|_{*}=\sup_{\boldsymbol{u}}\{\langle\boldsymbol{u},\boldsymbol{v}\rangle:\|\boldsymbol{u}\|\leq 1\}$. $\mathbb{E}[\boldsymbol{u}]$ means the expectation w.r.t. the underlying probability distribution of a random variable $\boldsymbol{u}$. The gradient of $f$ at $\boldsymbol{x}$ is denoted by $\nabla f(\boldsymbol{x})$.

Now, we describe our first-order stochastic black-box oracle. In our setting, we will query the stochastic oracle two times on each round $t=1,\ldots,T$, obtaining the noisy gradients $\boldsymbol{g}(\boldsymbol{x}_{t},\xi_{t})$ and $\boldsymbol{g}(\boldsymbol{x}_{t},\xi^{\prime}_{t})$. Note that, when convenient, we will refer to $\boldsymbol{g}(\boldsymbol{x}_{t},\xi_{t})$ and $\boldsymbol{g}(\boldsymbol{x}_{t},\xi^{\prime}_{t})$ as $\boldsymbol{g}_{t}$ and $\boldsymbol{g}_{t}^{\prime}$ respectively. We assume everywhere that our objective function is bounded from below and denote the infimum by $f^{\star}$, hence $f^{\star}>-\infty$. Also, we use $\mathbb{E}_{t}[\boldsymbol{u}]$ to denote the conditional expectation of $\boldsymbol{u}$ with respect to $\xi_{1},\ldots,\xi_{t-1},\xi^{\prime}_{1},\ldots,\xi^{\prime}_{t-1}$. Further, we will make use of the following assumptions:

H1: The noisy gradients at time $t$ are unbiased and independent given the past, that is

[TABLE]

H2: The noisy gradients $\boldsymbol{g}_{t}$ have finite variance with respect to the L2 norm:

[TABLE]

H3: The noisy gradients have bounded norm: $\|\boldsymbol{g}(\boldsymbol{x}_{t},\xi_{t})\|\leq L,\ \|\boldsymbol{g}(\boldsymbol{x}_{t},\xi^{\prime}_{t})\|\leq L$.

H4: The function $f:\mathbb{R}^{d}\rightarrow\mathbb{R}$ is $M$-smooth, that is $f$ is differentiable and $\forall\boldsymbol{x}_{1},\boldsymbol{x}_{2}\in\mathbb{R}^{d}$ we have $\|\nabla f(\boldsymbol{x}_{1})-\nabla f(\boldsymbol{x}_{2})\|\leq M\|\boldsymbol{x}_{1}-\boldsymbol{x}_{2}\|$. Note that (H4), for all $\boldsymbol{x},\boldsymbol{y}\in\mathbb{R}^{d}$, implies (Nesterov, 2003, Lemma 1.2.3)

[TABLE]

We will also consider the Polyak-Łojasiewicz (PL) condition (Karimi et al., 2016), a much weaker version of strong convexity. The PL condition does not require convexity, but is still sufficient for showing a global linear convergence rate for gradient descent.

H5: A differentiable function $f$ satisfies the PL condition if for some $\mu>0$

[TABLE]

4 Surrogate Losses

Consider using SGD with non-convex $M$-smooth losses, using a fixed stepsize $0<\eta\leq\frac{1}{M}$ and starting from an initial point $\boldsymbol{x}_{1}$. Assuming all the variances are bounded by $\sigma^{2}$, it is well-known that we obtain the following convergence rate (Ghadimi & Lan, 2013)

[TABLE]

where $k$ is a uniform random variable between $1$ and $T$. From the above, it is immediate to see that we need a stepsize of the form $O(\min(\tfrac{\sqrt{f(\boldsymbol{x}_{1})-f^{\star}}}{\sigma\sqrt{T}},\tfrac{1}{M}))$ to have the best worst case convergence of $O(\tfrac{1}{T}+\tfrac{\sigma}{\sqrt{T}})$. In words, this means that we get a faster rate, $O(\tfrac{1}{T})$, when there is no noise, and a slower rate, $O(\tfrac{\sigma}{\sqrt{T}})$, in the presence of noise.

However, we usually do not know the variance of the noise $\sigma$, which makes the above optimal tuning of the stepsize difficult to achieve in practice. Even worse, the variance can change over time. For example, it may decrease over time if $f(\boldsymbol{x})=\mathbb{E}[f_{j}(\boldsymbol{x})]$ and each $f_{j}$ has zero gradient at the local optimum we are converging to. Moreover, even assuming the knowledge of the variance of the noise, the stepsizes proposed in Ghadimi & Lan (2013) assume the knowledge of the unknown quantity $f(\boldsymbol{x}_{1})-f^{\star}$.

One solution would be to obtain an explicit estimate of the variances of the noise, for example by applying some concentration inequality to the sample variance, and use it to set the stepsizes. This approach is suboptimal because it does not directly optimize the convergence rates, relying instead on a loose worst-case analysis. Instead, we propose to directly estimate the stepsizes that achieve the best convergence rate using an online learning algorithm. Our approach is simple and efficient: we introduce new surrogate (strongly)-convex losses that make the problem of learning the stepsizes a simple one-dimensional online convex optimization problem.

Our strategy uses the smoothness of the objective function to transform the problem of optimizing a non-convex objective function to the problem of optimizing a series of convex loss functions, which we solve by an online learning algorithm. Specifically, at each time $t$ define the surrogate loss $\ell_{t}:\mathbb{R}\rightarrow\mathbb{R}$ as

[TABLE]

where $\boldsymbol{g}(\boldsymbol{x}_{t},\xi_{t})$ and $\boldsymbol{g}(\boldsymbol{x}_{t},\xi^{\prime}_{t})$ are the noisy stochastic gradients received from the black-box oracle at time $t$. It is clear that $\ell_{t}$ is a convex function. Moreover, the following key result shows that these surrogate losses upper bound the expected decrease of the function value $f$.

Theorem 1**.**

Assume (H4) holds and $\eta_{t}$ is independent from $\xi_{j}$ and $\xi^{\prime}_{j}$ for $j\geq t$. Then, for the SGD update in (1), we have

[TABLE]

Proof.

The $M$-smoothness of $f$ gives us:

[TABLE]

Now observe that $\nabla f(\boldsymbol{x}_{t})=\mathbb{E}_{t}\left[\boldsymbol{g}(\boldsymbol{x}_{t},\xi^{\prime}_{t})\right]$, so that

[TABLE]

Putting all together, we have the stated inequality. ∎

Note that the assumption of the independence of $\eta_{t}$ from the stochasticity $\xi_{j}$ and $\xi^{\prime}_{j}$ of the current step is essential according to Li & Orabona (2019).

The theorem tells us that if we want to decrease the function $f$, we might instead try to minimize the convex surrogate losses $\ell_{t}$. In the following, we build up on this intuition to design an online learning procedure that adapts the stepsizes of SGD to achieve the optimal convergence rate.

5 SGDOL: Adaptive Stepsizes with FTRL

The surrogate losses allow us to design an online convex optimization procedure to learn the optimal stepsizes. We call this procedure Stochastic Gradient Descent with Online Learning (SGDOL) and the pseudocode is in Algorithm 1. Remind that $\boldsymbol{g}(\boldsymbol{x}_{t},\xi_{t})$ and $\boldsymbol{g}(\boldsymbol{x}_{t},\xi^{\prime}_{t})$ are referred to as $\boldsymbol{g}_{t}$ and $\boldsymbol{g}_{t}^{\prime}$ respectively for convenience. In each round, the stepsizes are chosen by an online learning algorithm $\mathcal{A}$ fed with the surrogate losses $\ell_{t}$. The online learning algorithm aims at minimizing the regret: the difference between the cumulative sum of the losses incurred by the algorithm in each round, and the cumulative losses w.r.t any fixed point $\eta$ (especially the one giving the smallest losses in hindsight). In formulas, for a 1-dimensional online convex optimization problem, the regret is defined as

[TABLE]

A small regret with respect to the optimal choice of $\eta$ means that the losses of the algorithm are not too big compared to the best achievable losses from using a fixed point. In turn, this implies that the stepsizes chosen by the online algorithm will not be too far from the optimal (unknown) stepsize.

Employing SGDOL, we can prove the following Theorem.

Theorem 2**.**

Assume (H1, H2, H4) to hold. Then, for any $\eta>0$, SGDOL in Algorithm 1 satisfies

[TABLE]

Proof.

Summing the inequality in Theorem 1 from 1 to $T$:

[TABLE]

Using the fact that

[TABLE]

we have the stated bound. ∎

The only remaining ingredient for SGDOL is to decide an online learning procedure. Given that the surrogate losses are strongly convex, we can use a Follow The Regularized Leader (FTRL) algorithm (Shalev-Shwartz, 2007; Abernethy et al., 2008, 2012; McMahan, 2017). Note that this is not the only possibility, e.g. we could even use an optimistic FTRL algorithm that achieves even smaller regret (Mohri & Yang, 2016). However, FTRL is enough to show the potential of our surrogate losses. In an online learning game in which we receive the convex losses $\ell_{t}$, FTRL constructs the predictions $\boldsymbol{v}_{t}$ by solving the optimization problem

[TABLE]

where $r:\mathbb{R}^{d}\rightarrow\mathbb{R}$ is a regularization function. We can upper bound the regret of FTRL with the following theorem.

Theorem 3**.**

(McMahan, 2017)* Suppose $r$ is chosen such that $h_{t}=r+\sum_{i=1}^{t}\ell_{i}$ is 1-strongly-convex w.r.t. some norm $\|\cdot\|_{(t)}$. Then, choosing any $\boldsymbol{g}_{t}\in\partial\ell_{t}(\boldsymbol{x}_{t})$ on each round, for any $\boldsymbol{x}^{\star}\in\mathbb{R}^{d}$ and for any $T>0$,*

[TABLE]

where $\|\cdot\|_{(t),\star}$ is the dual norm of $\|\cdot\|_{(t)}$.

We can now put all together and obtain a convergence rate guarantee for SGDOL.

Theorem 4**.**

By choosing $r(\eta)=\frac{M\alpha}{2}\left(\eta-\frac{1}{M}\right)^{2}+\mathcal{I}\left(\eta\in\left[0,\frac{2}{M}\right]\right)$ with $\alpha>0$, assuming (H1 - H4), and using FTRL, Algorithm 2, in Algorithm 1, for an uniformly randomly picked $\boldsymbol{x}_{k}$ from $\boldsymbol{x}_{1},\ldots,\boldsymbol{x}_{T}$, we have:

[TABLE]

Before proving this theorem, we make some observations.

The FTRL update gives us a very simple strategy to calculate the stepsizes $\eta_{t}$. In particular, the FTRL update has a closed form:

[TABLE]

Note that this update can be efficiently computed by keeping track of the quantities $\sum^{t-1}_{j=1}\langle\boldsymbol{g}_{j},\boldsymbol{g}^{\prime}_{j}\rangle$ and $\sum^{t-1}_{j=1}\|\boldsymbol{g}_{j}\|^{2}$.

While the computational complexity of calculating $\eta_{t}$ by FTRL is negligible, SGDOL requires two unbiased gradients per step. This increases the computational complexity with respect to a plain SGD procedure by a factor of two.

The value of $\alpha$ affects how fast $\eta_{t}$ deviates from its initial value $\frac{1}{M}$. Although Theorem 4 shows that a too small $\alpha$ would blow up the log factor, it also indicates setting $\alpha$ to be comparable with $M(f(x_{1})-f(x^{*}))$ or smaller would suffice for not inducing a major influence on the convergence rate. In fact, preliminary experiments have shown that $\alpha$ has no notable influence on performance so long as it is comparable to M or smaller.

We can now prove the convergence rate in Theorem 4. For the proof, we need the following technical lemma.

Lemma 5**.**

Let $h:[0,+\infty)\rightarrow[0,+\infty)$ be a nonincreasing function, and $a_{i}\geq 0$ for $i=0,\cdots,T$. Then

[TABLE]

Proof.

Denote by $s_{t}=\sum_{i=0}^{t}a_{i}$.

[TABLE]

Summing over $t=1,\cdots,T$, we have the stated bound. ∎

Proof of Theorem 4.

As $\ell_{t}^{\prime\prime}(\eta)=M\|\boldsymbol{g}_{t}\|^{2}$, $h_{t}=r+\sum_{i=1}^{t}\ell_{i}$ is 1-strongly-convex w.r.t. the norm $\sqrt{M\left(\alpha+\sum^{t}_{s=1}\|\boldsymbol{g}_{s}\|^{2}\right)}\|\cdot\|$.

Applying Theorem 4, we get that, for any $\eta\in\left[0,\frac{2}{M}\right]$,

[TABLE]

Now observe that

[TABLE]

where in the third line of which we used the Cauchy-Schwarz inequality and $\eta_{t}\leq\frac{2}{M}$. Hence, we get

[TABLE]

where in the second inequality we used Lemma 5.

Put the last inequality above back into Theorem 2 yields

[TABLE]

Denote $A\triangleq\sum^{T}_{t=1}\mathbb{E}\left[\|\nabla f(\boldsymbol{x}_{t})\|^{2}\right]$, we can transform the above into a quadratic inequality of $\eta$:

[TABLE]

Choosing $\eta$ as the minimizer of the right hand side: $\eta^{*}=\frac{\alpha+A}{M(\alpha+A)+M\sum^{T}_{t=1}\mathbb{E}[\sigma_{t}^{2}]}$ (which satisfies $\eta^{*}\leq\frac{2}{M}$) gives us

[TABLE]

Solving this quadratic inequality of $A$ yields

[TABLE]

By taking an $\boldsymbol{x}_{k}$ from $\boldsymbol{x}_{1},\ldots,\boldsymbol{x}_{T}$ randomly, we get:

[TABLE]

that completes the proof. ∎

Polyak-Łojasiewicz Condition.

When we assume in addition that the objective function satisfies the Polyak-Łojasiewicz Condition (Karimi et al., 2016) (H5), we can get the linear rate in the noiseless case.

Theorem 6**.**

Choosing $r(\eta)=\frac{M\alpha}{2}\left(\eta-\frac{1}{M}\right)^{2}+\mathcal{I}\left(\eta\in\left[0,\frac{2}{M}\right]\right)$ with $\alpha>0$, assume (H1 - H5), and that $\boldsymbol{g}(\boldsymbol{x}_{t},\xi_{t})=\boldsymbol{g}(\boldsymbol{x}_{t},\xi^{\prime}_{t})=\nabla f(\boldsymbol{x}_{t})$ for all $t$ (i.e. there is no noise). Then feeding Algorithm 2 into Algorithm 1, yields:

[TABLE]

Proof.

From the update rule of $\eta_{t}$, we have that when there is no noise, $\eta_{t}=\frac{1}{M}$ all the time, thus:

[TABLE]

6 Adapting Per-coordinate Stepsizes

In the previous Section, we have shown how to use the surrogate loss functions to adapt a stepsize. Another common strategy in practice is to use a per-coordinate stepsize. This kind of scheme is easily incorporated into our framework and we show that it can provide improved adaptivity to per-coordinate variances.

Specifically, we consider $\boldsymbol{\eta}_{t}$ now to be a vector in $\mathbb{R}^{d}$, $\boldsymbol{\eta}_{t}=(\eta_{t,1},\dots,\eta_{t,d})$ and use the update $\boldsymbol{x}_{t+1}=\boldsymbol{x}_{t}-\boldsymbol{\eta}_{t}\boldsymbol{g}_{t}$ where $\boldsymbol{\eta}_{t}\boldsymbol{g}_{t}$ now indicates coordinate-wise product $(\eta_{t,1}g_{t,1},\dots,\eta_{t,d}g_{t,d})$. Then we define

[TABLE]

To take advantage of this scenario, we need more detail about the variance, which we encapsulate in the following assumption:

H2’: The noisy gradients $\boldsymbol{g}_{t}$ have finite variance in each coordinate:

[TABLE]

Note that this assumption is not actually stronger than (H2) because we can define $\sigma_{t}^{2}=\sum_{i=1}^{d}\sigma_{t,i}^{2}$. This merely provides finer-grained variable names.

Also, we make the assumption:

H3’: The noisy gradients have bounded coordinate values:

[TABLE]

Now the exact same argument as for Theorem 2 yields:

Theorem 7**.**

Assume (H4) and the two noisy gradients in each round $t$ to satisfy (H1) and (H2’). Then, for any $\eta\in\mathbb{R}^{d}$ with $\eta_{i}>0$ for all $i$, the per-coordinate variant of Algorithm 1 obtains

[TABLE]

With this Theorem in hand, once again all that remains is to choose the online learning algorithm. To this end, observe that we can write $\ell_{t}(\eta)=\sum_{i=1}^{d}\ell_{t,i}(\eta_{i})$ where

[TABLE]

Thus, we can take our online learning algorithm to be a per-coordinate instantiation of Algorithm 2, and the total regret is simply the sum of the per-coordinate regrets. Each per-coordinate regret can be analyzed in exactly the same way as Algorithm 2, leading to

[TABLE]

From these inequalities we can make a per-coordinate bound on the gradient magnitudes. In words, the coordinates which have smaller variances $\sigma^{2}_{t,i}$ achieve smaller gradient values faster than coordinates with larger variances. Further, we preserve adaptivity to the full variance $\sum^{T}_{t=1}\mathbb{E}[\sigma_{t}^{2}]$ in the rate of decrease of $\|\nabla f(x)\|^{2}$.

Theorem 8**.**

Assume (H1, H2’, H3’, H4). Suppose we run a per-coordinate variant of Algorithm 1, with regularizer $r(\eta_{i})=\frac{M\alpha}{2}\left(\eta_{i}-\frac{1}{M}\right)^{2}+\mathcal{I}\left(\eta_{i}\in\left[0,\frac{2}{M}\right]\right)$ in each coordinate with $\alpha>0$. Then, for each $i\in\{1,\dots,d\}$, we have

[TABLE]

Further, with $\sigma_{t}=\sum_{t=1}^{T}\sigma_{t,i}^{2}$ it also holds

[TABLE]

Proof.

The proof is nearly identical to that of Theorem 4. We have

[TABLE]

Let $A_{i}\triangleq\mathbb{E}[\sum_{t=1}^{T}\nabla f(\boldsymbol{x}_{t})_{i}^{2}]$ and choosing $\eta_{i}$ by the same strategy in Theorem 4 as $\frac{\alpha+A_{i}}{M(\alpha+A_{i}+\sum_{t=1}^{T}\mathbb{E}[\sigma_{t,i}^{2}])}$ to obtain

[TABLE]

Now, the first statement of the Theorem follows by observing that each term on the LHS is non-negative so that the sum can be lower-bounded by any individual term. For the second statement, define

[TABLE]

so that $\sum_{i=1}^{d}Q_{i}\leq Q$. By the quadratic formula and definition of $Q_{i}$, we have

[TABLE]

Thus,

[TABLE]

From which the second statement follows. ∎

7 Experiments

SGD is widely known to enjoy good empirical properties, but our learning rate schedule is very unique, so to validate our theoretical findings, we experiment on fitting a classification model on the adult (a9a) dataset from the LibSVM website (Chang & Lin, 2001). The objective function is

[TABLE]

where $\phi(\theta)=\frac{\theta^{2}}{1+\theta^{2}}$, and $(\boldsymbol{a}_{i},y_{i})$ are the couples feature vector/label. The loss function $\phi$ is non-convex, 1-Lipschitz and 2-smooth w.r.t. the $\ell_{2}$ norm.

We consider the minimization problem with respect to all training samples. Also, as the dataset is imbalanced towards the group with annual income less than 50K, we subsample that group to balance the dataset, which results in 15682 samples with 123 features each. In addition, we append a constant element to each sample feature vector to introduce a constant bias. $\boldsymbol{x}_{1}$ is initialized to be all zeros. For each setting, we repeat the experiment independently but with the same initialization for 5 times, and plot the average of the relevant quantities. In this setting, the noise on the gradient is generated by the use of minibatches.

We compare SGDOL with AdaGrad (Duchi et al., 2010), SGD, and Adam (Kingma & Ba, 2015) on three different minibatch sizes, namely different noise scales: using all samples, 50 i.i.d. samples, or 1 random sample for evaluating the gradient at a point. (Note that we adopt the scheme of using a single learning rate for all dimensions in SGDOL and AdaGrad thus the suffix ‘Global’.) The learning rates of each algorithm, except for SGDOL Global, are selected as the ones giving the best convergence rate when the full batch scheme, namely zero noise, is employed, and are shown in the legend. We take the reciprocal of SGD’s best learning rate as the parameter $M$ for SGDOL Global, and we set $\alpha=10$ without any tuning based on our discussion on the influence of $\alpha$ in Section 5. These parameters are then employed in other two noisy settings.

We report the results in Figure 1. In each column, the top plot shows $\mathbb{E}[\|\nabla f(\boldsymbol{x}_{k})\|^{2}]$ vs. number of iterations, whereas the bottom one is the per-round stepsizes on each case. Note that there is no per-round stepsize for Adam. The x-axis in all figures, and the y-axis in the top three are logarithmic.

As can be seen, the stepsize of SGDOL is the same as SGD at first, but gradually decreases automatically. Also, the larger the noise, the sooner the decreasing phase starts. The decrease of the learning rate makes the convergence of SGDOL possible. In particular, SGDOL recovers the performance of SGD in the noiseless case, while it allows convergence in the noisy cases through an automatic decrease of the stepsizes. AdaGrad also enjoys nice convergence, and is comparable to ours. In contrast, when noise exists, after reaching a proximity of a stationary point, SGD and Adam oscillates thereafter without converging, and the value it oscillates around depends on the variance of the noise. This underlines the superiority of the surrogate losses, rather than choosing a stepsize based on a worst-case convergence rate bound.

More experiments can be found in the Appendix.

8 Conclusions and Future Work

We have presented a novel way to cast the problem of adaptive stepsize selection for the stochastic optimization of smooth (non-convex) functions as an online convex optimization problem with a simple quadratic convex surrogate. The reduction goes through the use of novel surrogate convex losses. This framework allows us to import the rich literature of no-regret online algorithms to learn stepsizes on the fly. We exemplified the power of this method with the SGDOL algorithm which enjoys an optimal convergence guarantee for any level of noise, without the need to estimate the noise nor tune the stepsizes. Moreover, we recover linear convergence rates under the PL-condition. The overall price to pay is a factor of 2 in the computation of the gradients. We also presented a per-coordinate version of SGDOL that achieves faster convergence on the coordinates with less noise.

We feel that we have barely scratched the surface of what might be possible with these surrogate losses. Hence, future work will focus on extending their use to other scenarios. For example, we plan to use it in locally private SGD algorithms where additional noise is added on the gradients to ensure privacy of the data (Song et al., 2013). We are also interested in investigating whether adding convexity would give us better results to recover SGD’s performance. Another potential direction is to eliminate the need of knowing $M$, e.g. by automatically adapting to it on the fly.

9 Acknowledgment

This material is based upon work supported by the National Science Foundation under grant no. 1740762 “Collaborative Research: TRIPODS Institute for Optimization and Learning”.

10 Appendix

10.1 2D Rosenbrock Function

The popular 2-D Rosenbrock benchmark (Rosenbrock, 1960) takes the form:

[TABLE]

It is non-convex and has one global minimum at $x=y=1$.

To add stochasticity, we apply additive white Gaussian noise to each gradient. To have a robust estimate of the performance, we repeat each experiment independently with the same parameters for 40 times and take the average.

We compare the performance of SGDOL Global with a bunch of popular adaptive optimization algorithms on the Rosenbrock function with 3 levels of added noise: zero noise, small noise ($\sigma=0.2$), and large noise ($\sigma=5$). The competitors are: SGD, AdaGrad Global (Duchi et al., 2010) (with one learning rate for all dimensions), AdaGrad Coordinate (Duchi et al., 2010) (with one learning rate for each dimension), Adam (Kingma & Ba, 2015), RMSProp (Tieleman & Hinton, 2012), AdaDelta (Zeiler, 2012), AMSGrad (Reddi et al., 2018), and Hypergradient (Baydin et al., 2018). Also, we test the performance of the stepsize proposed by Ghadimi & Lan (2013), denoted by SGD GL, given that in this synthetic experiment we know all the relevant quantities. We stress the fact that in the real-world setting this kind of stepsize cannot be used. We select the stepsize of all optimization algorithms except for SGDOL Global and SGD GL to be the one giving best convergence rate when running on the objective function with zero noise added. We choose $M$ of SGDOL to be the reciprocal of SGD’s best learning rate which happens to be very close to the smoothness at the optimal point, 1002. We set $x_{1}=y_{1}=0$.

In Figure 2, the top plots show $\mathbb{E}[\|\nabla f(\boldsymbol{x}_{k})\|^{2}]$ vs. number of iterations, the middle ones reflect the curve of the optimality gap $f(\boldsymbol{x}_{t})-f^{*}$ at each round $t$ since we know $f^{*}=0$, whereas the bottom ones are the per-round stepsizes on each case. Note that there is no per-round stepsize for Adam. In Figure 3, the top plots show $\mathbb{E}[\|\nabla f(\boldsymbol{x}_{k})\|^{2}]$ vs. number of iterations, and the bottom ones reflect the curve of the optimality gap $f(\boldsymbol{x}_{t})-f^{*}$ at each round $t$. The x-axis in all figures are logarithmic, and the y-axis in all figures except for those showing stepsizes are logarithmic.

The behavior of both the curve of $\mathbb{E}[\|\nabla f(\boldsymbol{x}_{k})\|^{2}]$ vs. number of iterations and the curve of stepsizes are similar to Figure 1. And the behavior of the curve of optimality gap is similar to that of the curve of $\mathbb{E}[\|\nabla f(\boldsymbol{x}_{k})\|^{2}]$ vs. number of iterations.

10.2 Other Results for Fitting a Non-Linear Classification Model

Here we show the results for comparison between SGDOL Global and other optimization algorithms listed in the above subsection but applied to the classification task introduced in Section 7. Note that here we don’t know $f^{*}$ so we don’t report the curve of the optimality gap. The comparison shown in Figure 4 is similar to what is reported in Figure 1.