Adaptive Restart of the Optimized Gradient Method for Convex Optimization

TL;DR

This paper explores adaptive restart techniques to accelerate the optimized gradient method, improving convergence rates for convex optimization problems, especially when parameters are unknown or the iterates are in strongly convex regions.

Contribution

It introduces a heuristic adaptive restart scheme for the optimized gradient method, enhancing its convergence speed for convex and strongly convex problems.

Findings

Adaptive restart accelerates the optimized gradient method.

Numerical experiments confirm improved convergence rates.

Helpful for nonsmooth composite convex functions.

Abstract

First-order methods with momentum such as Nesterov's fast gradient method are very useful for convex optimization problems, but can exhibit undesirable oscillations yielding slow convergence rates for some applications. An adaptive restarting scheme can improve the convergence rate of the fast gradient method, when the parameter of a strongly convex cost function is unknown or when the iterates of the algorithm enter a locally strongly convex region. Recently, we introduced the optimized gradient method, a first-order algorithm that has an inexpensive per-iteration computational cost similar to that of the fast gradient method, yet has a worst-case cost function rate that is twice faster than that of the fast gradient method and that is optimal for large-dimensional smooth convex problems. Building upon the success of accelerating the fast gradient method using adaptive restart, this paper investigates similar heuristic acceleration of the optimized gradient method. We first derive a new first-order method that resembles the optimized gradient method for strongly convex quadratic problems with known function parameters, yielding a linear convergence rate that is faster than that of the analogous version of the fast gradient method. We then provide a heuristic analysis and numerical experiments that illustrate that adaptive restart can accelerate the convergence of the optimized gradient method. Numerical results also illustrate that adaptive restart is helpful for a proximal version of the optimized gradient method for nonsmooth composite convex functions.

Tables

Table 1. Table 1: Accelerated First-order Methods for Smooth Convex Problems

Method	$\alpha$	${\beta }_k$	${\gamma }_k$	Worst-case Rate
GM	$\frac{1}{L}$	$0$	$0$	$f\left({𝒚}_k\right)-f\left({𝒙}_{\ast }\right)\le \frac{L{\Vert {𝒙}_0-{𝒙}_{\ast }\Vert }^2}{4k+2}$ drori:14:pof
FGM nesterov:83:amf	$\frac{1}{L}$	$\frac{t_k-1}{t_{k+1}}$	$0$	$f\left({𝒚}_k\right)-f\left({𝒙}_{\ast }\right)\le \frac{L{\Vert {𝒙}_0-{𝒙}_{\ast }\Vert }^2}{2t_{k-1}^2}\le \frac{2L{\Vert {𝒙}_0-{𝒙}_{\ast }\Vert }^2}{{\left(k+1\right)}^2}$ beck:09:afi
				$f\left({𝒙}_k\right)-f\left({𝒙}_{\ast }\right)\le \frac{L{\Vert {𝒙}_0-{𝒙}_{\ast }\Vert }^2}{2t_k^2}\le \frac{2L{\Vert {𝒙}_0-{𝒙}_{\ast }\Vert }^2}{{\left(k+2\right)}^2}$ kim:16:ofo
OGM′ kim:17:otc	$\frac{1}{L}$	$\frac{t_k-1}{t_{k+1}}$	$\frac{t_k}{t_{k+1}}$	$f\left({𝒚}_k\right)-f\left({𝒙}_{\ast }\right)\le \frac{L{\Vert {𝒙}_0-{𝒙}_{\ast }\Vert }^2}{4t_{k-1}^2}\le \frac{L{\Vert {𝒙}_0-{𝒙}_{\ast }\Vert }^2}{{\left(k+1\right)}^2}$ kim:17:otc
OGM kim:16:ofo	$\frac{1}{L}$	$\frac{{\theta }_k-1}{{\theta }_{k+1}}$	$\frac{{\theta }_k}{{\theta }_{k+1}}$	$f\left({𝒙}_N\right)-f\left({𝒙}_{\ast }\right)\le \frac{L{\Vert {𝒙}_0-{𝒙}_{\ast }\Vert }^2}{2{\theta }_N^2}\le \frac{L{\Vert {𝒙}_0-{𝒙}_{\ast }\Vert }^2}{{\left(N+1\right)}^2}$ kim:16:ofo
Parameters
$t_0=1,t_k=\frac{1}{2}\left(1+\sqrt{1+4t_{k-1}^2}\right),k=1,\dots ,$

Table 2. Table 2: Accelerated First-order Methods (with γ k = 0 subscript 𝛾 𝑘 0 \gamma_{k}=0 ) for Smooth and Strongly Convex Problems (The worst-case rates also apply to μ 2 ​ ‖ 𝒚 k − 𝒙 ∗ ‖ 2 𝜇 2 superscript norm subscript 𝒚 𝑘 subscript 𝒙 2 \frac{\mu}{2}||\bm{y}_{k}-\bm{x}_{*}||^{2} due to the strong convexity ( 2 ).)

Method	$\alpha$	${\beta }_k$	Worst-case Rate
GM	$\frac{1}{L}$	$0$	$f\left({𝒚}_k\right)-f\left({𝒙}_{\ast }\right)\le {\left(1-\frac{2\mu }{1+q}\right)}^k\frac{L{\Vert {𝒙}_0-{𝒙}_{\ast }\Vert }^2}{2}$ nesterov:04
GM-$q$	$\frac{2}{\mu +L}$	$0$	$f\left({𝒚}_k\right)-f\left({𝒙}_{\ast }\right)\le {\left(\frac{1-q}{1+q}\right)}^{2k}\frac{L{\Vert {𝒙}_0-{𝒙}_{\ast }\Vert }^2}{2}$ nesterov:04
FGM-$q$ nesterov:04	$\frac{1}{L}$	$\frac{1-\sqrt{q}}{1+\sqrt{q}}$	$f\left({𝒚}_k\right)-f\left({𝒙}_{\ast }\right)\le {\left(1-\sqrt{q}\right)}^k\frac{\left(1+q\right)L{\Vert {𝒙}_0-{𝒙}_{\ast }\Vert }^2}{2}$ nesterov:04

Table 3. Table 3: Optimally tuned coefficients ( α , β , γ ) 𝛼 𝛽 𝛾 (\alpha,\beta,\gamma) of GM- q 𝑞 q , FGM- q 𝑞 q , FGM ′ - q 𝑞 q , and OGM- q 𝑞 q , and their spectral radius ρ ​ ( 𝑻 ​ ( α , β , γ ) ) 𝜌 𝑻 𝛼 𝛽 𝛾 \rho(\bm{T}(\alpha,\beta,\gamma))  ( 9 ). These optimal coefficients result from solving ( 12 ) with the shaded coefficients fixed.

Method	$\alpha$	$\beta$	$\gamma$	$\rho \left(𝑻(\alpha ,\beta ,\gamma )\right)$
GM-$q$	$\frac{2}{\mu +L}$	$0$	$0$	$\frac{1-q}{1+q}$
FGM-$q$ nesterov:04	$\frac{1}{L}$	$\frac{1-\sqrt{q}}{1+\sqrt{q}}$	$0$	$1-\sqrt{q}$
FGM′-$q$ lessard:16:aad	$\frac{4}{\mu +3L}$	$\frac{\sqrt{3+q}-2\sqrt{q}}{\sqrt{3+q}+2\sqrt{q}}$	$0$	$1-\frac{2\sqrt{q}}{\sqrt{3+q}}$
OGM-$q$	$\frac{1}{L}$	$\frac{{\left(2+q-\sqrt{q^2+8q}\right)}^2}{4\left(1-q\right)}$	$\frac{2+q-\sqrt{q^2+8q}}{2}$	$\frac{2+q-\sqrt{q^2+8q}}{2}$

Equations

$$\underset{\mathbf{x}\in {\mathbb{R}}^d}{\min }$$

$$||\nabla f(\mathbf{x})-\nabla f(\mathbf{y})||\le L||\mathbf{x}-\mathbf{y}||,\forall \mathbf{x},\mathbf{y}\in {\mathbb{R}}^d,$$

$$f(\mathbf{x})\ge f(\mathbf{y})+⟨\nabla f(\mathbf{y}),\mathbf{x}-\mathbf{y}⟩+\frac{\mu }{2}||\mathbf{x}-\mathbf{y}|{|}^2,\forall \mathbf{x},\mathbf{y}\in {\mathbb{R}}^d.$$

$$f(\mathbf{x})=\frac{1}{2}{\mathbf{x}}^{\top }\mathbf{Q}\mathbf{x}-{\mathbf{p}}^{\top }\mathbf{x}\in {\mathcal{F}}_{\mu ,L}({\mathbb{R}}^d)$$

$${\mathbf{\xi }}_{k+1}-{\mathbf{\xi }}_{\ast }=\mathbf{T}(\alpha ,\beta ,\gamma )({\mathbf{\xi }}_k-{\mathbf{\xi }}_{\ast }),$$

$$\displaystyle\bm{T}(\alpha,\beta,\gamma):=\bigg{[}\begin{array}[]{cc}(1+\beta)(\bm{I}-\alpha\bm{Q})-\gamma\alpha\bm{Q}&-\beta(\bm{I}-\alpha\bm{Q})\\ \bm{I}&\bm{0}\end{array}\bigg{]}\in\mathbb{R}^{2d\times 2d}$$

$$||{\mathbf{\xi }}_{k+1}-{\mathbf{\xi }}_{\ast }|{|}^2\le (\rho (\mathbf{T}(\alpha ,\beta ,\gamma ))+ϵ{)}^{2k}||{\mathbf{\xi }}_1-{\mathbf{\xi }}_{\ast }|{|}^2.$$

$$\rho (\mathbf{T}(\alpha ,\beta ,\gamma ))=\underset{\mu \le \lambda \le L}{\max }\rho ({\mathbf{T}}_{\lambda }(\alpha ,\beta ,\gamma )),$$

$$\rho ({\mathbf{T}}_{\lambda }(\alpha ,\beta ,\gamma ))=\max \{|r_1(\alpha ,\beta ,\gamma ,\lambda )|,|r_2(\alpha ,\beta ,\gamma ,\lambda )|\}$$

$$=$$

$$r^2-((1+\beta )(1-\alpha \lambda )-\gamma \alpha \lambda )r+\beta (1-\alpha \lambda ),$$

$${\text{operator@font}argmin}_{\alpha ,\beta ,\gamma }\rho (\mathbf{T}(\alpha ,\beta ,\gamma ))={\text{operator@font}argmin}_{\alpha ,\beta ,\gamma }\max \{\rho ({\mathbf{T}}_{\mu }(\alpha ,\beta ,\gamma )),\rho ({\mathbf{T}}_L(\alpha ,\beta ,\gamma ))\}.$$

$$\Delta (1/L,\beta ,\gamma ,\mu )=((1+\beta )(1-q)-\gamma q{)}^2-4\beta (1-q)$$

$$=$$

$${\beta }^{\star }(\gamma ):={\left(1-\sqrt{q(1+\gamma )}\right)}^2/(1-q),$$

$${\gamma }^{\star }:={\text{operator@font}argmin}_{\gamma }\max \left\{|1-\sqrt{q(1+\gamma )}|,|\gamma |\right\}.$$

$${\beta }^{\star }:={\beta }^{\star }({\gamma }^{\star })=\frac{{\left({\gamma }^{\star }\right)}^2}{1-q}=\frac{(2+q-\sqrt{q^2+8q}{)}^2}{4(1-q)},{\gamma }^{\star }=\frac{2+q-\sqrt{q^2+8q}}{2},$$

$$w_{k+2,i}$$

$$v_{k+2,i}$$

$$w_{1,i}=(1-{\lambda }_i/L)w_{0,i},v_{1,i}=((1+\beta +\gamma )(1-{\lambda }_i/L)-(\beta +\gamma ))v_{0,i}$$

$$w_{k,d}=0^k+c_d(-\gamma {)}^k,v_{k,d}=0^k+{\hat{c}}_d(-\gamma {)}^k,$$

$$c_d=0,{\hat{c}}_d=v_{0,d}\left(1+\beta /\gamma \right),$$

$${\beta }_i^{\star }(\gamma ):={\left(1-\sqrt{(1+\gamma ){\lambda }_i/L}\right)}^2/(1-{\lambda }_i/L).$$

$$w_{k,i}=c_{1,i}{r}_{1,i}^k+c_{2,i}{r}_{2,i}^k,v_{k,i}={\hat{c}}_{1,i}{r}_{1,i}^k+{\hat{c}}_{2,i}{r}_{2,i}^k,$$

$$r_i^{\star }(\gamma ):=1-\sqrt{(1+\gamma ){\lambda }_i/L},$$

$$w_{k,i}=c_i{\left(\sqrt{\beta (1-{\lambda }_i/L)}\right)}^k\text{operator@font}cos\left(k{\psi }_i(\beta ,\gamma \right)-{\delta }_i),$$

$$v_{k,i}={\hat{c}}_i{\left(\sqrt{\beta (1-{\lambda }_i/L)}\right)}^k\text{operator@font}cos\left(k{\psi }_i(\beta ,\gamma \right)-{\hat{\delta }}_i),$$

$${\psi }_i(\beta ,\gamma ):={\text{operator@font}cos}^{-1}\left(\left((1+\beta )(1-{\lambda }_i/L)-\gamma {\lambda }_i/L\right)/\left(2\sqrt{\beta (1-{\lambda }_i/L)}\right)\right),$$

$$f({\mathbf{x}}_{j,k})-f({\mathbf{x}}_{\ast })\le \frac{L||{\mathbf{x}}_{j,0}-{\mathbf{x}}_{\ast }|{|}^2}{k^2}\le \frac{2L}{\mu k^2}(f({\mathbf{x}}_{j,0})-f({\mathbf{x}}_{\ast })).$$

$$f({\mathbf{y}}_{k+1})>f({\mathbf{y}}_k),$$

$$⟨-\nabla f({\mathbf{x}}_k),{\mathbf{y}}_{k+1}-{\mathbf{y}}_k⟩<0.$$

$$⟨\nabla f({\mathbf{x}}_k),\nabla f({\mathbf{x}}_{k-1})⟩<0.$$

Full text

1 Introduction

The computational expense of first-order methods depends only mildly on the problem dimension, so they are attractive for solving large-dimensional optimization problems cevher:14:cof . In particular, Nesterov’s fast gradient method (FGM) nesterov:83:amf ; nesterov:04 ; beck:09:afi is used widely because it has a worst-case cost function rate that is optimal up to constant for large-dimensional smooth convex problems nesterov:04 . In addition, for smooth and strongly convex problems where the strong convexity parameter is known, a version of FGM has a linear convergence rate nesterov:04 that improves upon that of a standard gradient method. However, without knowledge of the function parameters, conventional FGM does not guarantee a linear convergence rate.

When the strong convexity parameter is unknown, a simple adaptive restarting scheme odonoghue:15:arf for FGM heuristically improves its convergence rate (see also giselsson:14:mar ; su:16:ade for theory and cevher:14:cof ; muckley:15:fpm ; monteiro:16:aaa for applications). In addition, adaptive restart is useful even when the function is only locally strongly convex near the minimizer odonoghue:15:arf . First-order methods are known to be suitable when only moderate solution accuracy is required, and adaptive restart can help first-order methods achieve medium to high accuracy.

Recently we proposed the optimized gradient method (OGM) kim:16:ofo (built upon drori:14:pof ) that has efficient per-iteration computation similar to FGM yet that exactly achieves the optimal worst-case rate for decreasing a large-dimensional smooth convex function among all first-order methods drori:17:tei . (See kim:16:gto-arxiv ; kim:17:otc ; taylor:17:ewc for further analysis and extensions of OGM.) This paper examines a general class of accelerated first-order methods that includes a gradient method (GM), FGM, and OGM for strongly convex quadratic functions, and develops an OGM variant, named OGM-$q$, that provides a linear convergence rate that is faster than that of the analogous version of FGM. The analysis reveals that, like FGM odonoghue:15:arf , OGM may exhibit undesirable oscillating behavior in some cases. Building on the quadratic analysis and the adaptive restart scheme of FGM in odonoghue:15:arf , we propose an adaptive restart scheme that heuristically accelerates the convergence rate of OGM when the function is strongly convex or even when it is only locally strongly convex. This restart scheme circumvents the oscillating behavior. Numerical results illustrate that the proposed OGM with restart performs better than FGM with restart in odonoghue:15:arf .

Sec. 2 reviews first-order methods for convex problems such as GM, FGM, and OGM. Sec. 3 analyzes a general class of accelerated first-order methods that includes GM, FGM, and OGM for strongly convex quadratic problems, and proposes a new OGM variant with a fast linear convergence rate. Sec. 4 suggests an adaptive restart scheme for OGM using the quadratic analysis in Sec. 3. Sec. 5 illustrates the proposed adaptive version of OGM that we use for numerical experiments on various convex problems in Sec. 6, including nonsmooth composite convex functions, and Sec. 7 concludes.

2 Problem and Methods

2.1 Smooth and Strongly Convex Problem

We first consider the smooth and strongly convex minimization problem:

[TABLE]

that satisfies the following smooth and strongly convex conditions:

• •

$f\;:\;\mathbb{R}^{d}\rightarrow\mathbb{R}$ has Lipschitz continuous gradient with Lipschitz constant $L>0$, i.e.,

[TABLE]

• •

$f$ is strongly convex with strong convexity parameter $\mu>0$, i.e.,

[TABLE]

We let $\mathcal{F}_{\mu,L}(\mathbb{R}^{d})$ denote the class of functions $f$ that satisfy the above two conditions hereafter, and let $\bm{x}_{*}$ denote the unique minimizer of $f$. We let $q:={\mu}/{L}$ denote the reciprocal of the condition number of a function $f\in\mathcal{F}_{\mu,L}(\mathbb{R}^{d})$. We also let $\mathcal{F}_{0,L}(\mathbb{R}^{d})$ denote the class of smooth convex functions $f$ that satisfy the above two conditions with $\mu=0$, and let $\bm{x}_{*}$ denote a minimizer of $f$.

Some algorithms discussed in this paper require knowledge of both $\mu$ and $L$, but in many cases estimating $\mu$ is challenging compared to computing $L$.111 For some applications even estimating $L$ is expensive, and one must employ a backtracking scheme beck:09:afi or similar approaches. We assume $L$ is known throughout this paper. An estimate of $\mu$ could be found by a backtracking scheme as described in (nesterov:13:gmf, , Sec. 5.3).

Therefore, this paper focuses on the case where the parameter $\mu$ is unavailable while $L$ is available. Even without knowing $\mu$, the adaptive restart approach in odonoghue:15:arf and the proposed adaptive restart approach in this paper both exhibit linear convergence rates in strongly convex cases.

We next review known accelerated first-order methods for solving (M).

2.2 Review of Accelerated First-order Methods

This paper focuses on accelerated first-order methods (AFM) of the form shown in Alg. 1. The fast gradient method (FGM) nesterov:83:amf ; nesterov:04 ; beck:09:afi (with $\gamma_{k}=0$ in Alg. 1) accelerates the gradient method (GM) (with $\beta_{k}=\gamma_{k}=0$) using the momentum term $\beta_{k}(\bm{y}_{k+1}-\bm{y}_{k})$ with negligible additional computation. The optimized gradient method (OGM) kim:16:ofo ; kim:17:otc uses an over-relaxation term $\gamma_{k}(\bm{y}_{k+1}-\bm{x}_{k})=-\gamma_{k}\alpha\nabla f(\bm{x}_{k})$ for further acceleration.

Tables 1 and 2 summarize the standard choices of coefficients $(\alpha,\beta_{k},\allowbreak\gamma_{k})$ for GM, FGM, OGM in nesterov:83:amf ; nesterov:04 ; beck:09:afi ; kim:16:ofo ; kim:17:otc and their worst-case rates for smooth convex functions $\mathcal{F}_{0,L}(\mathbb{R}^{d})$ and smooth and strongly convex functions $\mathcal{F}_{\mu,L}(\mathbb{R}^{d})$ respectively. (Other choices can be found in nesterov:04 ; kim:16:gto-arxiv ; chambolle:15:otc .) For convenience hereafter, we use the names GM, GM-$q$, FGM, FGM-$q$, OGM, and OGM*′* to distinguish different choices of standard AFM coefficients in Tables 1 and 2.

The worst-case OGM rate kim:16:ofo in Table 1 is about twice faster than the FGM rate beck:09:afi and is optimal for first-order methods for the function class $\mathcal{F}_{0,L}(\mathbb{R}^{d})$ under the large-scale condition $d\geq N+1$ drori:17:tei . However, it is yet unknown which first-order methods provide an optimal worst-case linear convergence rate for the function class $\mathcal{F}_{\mu,L}(\mathbb{R}^{d})$; this topic is left as an interesting future work.222 Recently, vanscoy:18:tfk developed a new first-order method for known $q$ that is not in AFM class but achieves a linear worst-case rate $(1-\sqrt{q})^{2}$ for the decrease of a strongly convex function that is faster than the linear rate $(1-\sqrt{q})$ of FGM-$q$ in Table 2. Towards this direction, Sec. 3 studies AFM for strongly convex quadratic problems, leading to a new method named OGM-$q$ with a linear convergence rate that is faster than that of FGM-$q$. Sec. 4 uses this quadratic analysis to analyze an adaptive restart scheme for OGM.

3

Analysis of AFM for Quadratic Functions

This section analyzes the behavior of AFM for minimizing a strongly convex quadratic function. The quadratic analysis of AFM in this section is similar in spirit to the analyses of a heavy-ball method (polyak:87, , Sec. 3.2) and AFM with $\gamma_{k}=0$ (lessard:16:aad, , Appx. A) (odonoghue:15:arf, , Sec. 4).

In addition, Sec. 3.3 optimizes the coefficients of AFM for such quadratic functions, yielding a linear convergence rate that is faster than that of FGM-$q$. The resulting method, named OGM-$q$, requires the knowledge of $q$, and Sec. 3.4 shows that using OGM (and OGM*′*) in Table 1 instead (without the knowledge of $q$) will cause the OGM iterates to oscillate when the momentum is larger than a critical value. This analysis stems from the dynamical system analysis of AFM with $\alpha={1}/{L}$ and $\gamma_{k}=0$ in (odonoghue:15:arf, , Sec. 4).

3.1 Quadratic Analysis of AFM

This section considers minimizing a strongly convex quadratic function:

[TABLE]

where $\bm{Q}\in\mathbb{R}^{d\times d}$ is a symmetric positive definite matrix, $\bm{p}\in\mathbb{R}^{d}$ is a vector. Here, $\nabla f(\bm{x})=\bm{Q}\bm{x}-\bm{p}$ is the gradient, and $\bm{x}_{*}=\bm{Q}^{-1}\bm{p}$ is the optimum. The smallest and the largest eigenvalues of $\bm{Q}$ correspond to the parameters $\mu$ and $L$ of the function respectively. For simplicity in the quadratic analysis, we consider the version of AFM that has constant coefficients $(\alpha,\beta,\gamma)$.

Defining the vectors $\bm{\xi}_{k}:=(\bm{x}_{k}^{\top},\bm{x}_{k-1}^{\top})^{\top}\in\mathbb{R}^{2d}$ and $\bm{\xi}_{*}:=(\bm{x}_{*}^{\top},\bm{x}_{*}^{\top})^{\top}\in\mathbb{R}^{2d}$, and extending the analysis for AFM with $\gamma=0$ in (lessard:16:aad, , Appx. A), AFM has the following equivalent form for $k\geq 1$:

[TABLE]

where the system matrix $\bm{T}(\alpha,\beta,\gamma)$ of AFM is defined as

[TABLE]

for an identity matrix $\bm{I}\in\mathbb{R}^{d\times d}$. The sequence $\{\tilde{\bm{\xi}}_{k}:=(\bm{y}_{k}^{\top},\bm{y}_{k-1}^{\top})^{\top}\}_{k\geq 1}$ also satisfies the recursion (4), implying that (4) characterizes the behavior of both the primary sequence $\{\bm{y}_{k}\}$ and the secondary sequence $\{\bm{x}_{k}\}$ of AFM with constant coefficients.

The spectral radius $\rho(\bm{T}(\cdot))$ of matrix $\bm{T}(\cdot)$ determines the convergence rate of the algorithm. Specifically, for any $\epsilon>0$, there exists $K\geq 0$ such that $[\rho(\bm{T})]^{k}\leq||\bm{T}^{k}||\leq(\rho(\bm{T})+\epsilon)^{k}$ for all $k\geq K$, establishing the following worst-case rate:

[TABLE]

We next analyze $\rho(\bm{T}(\alpha,\beta,\gamma))$.

Considering the eigen-decomposition of $\bm{Q}$ in $\bm{T}(\cdot)$ as in (lessard:16:aad, , Appx. A), the spectral radius of $\bm{T}(\cdot)$ is:

[TABLE]

where for any eigenvalue $\lambda$ of matrix $\bm{Q}$ we define a matrix $\bm{T}_{\lambda}(\alpha,\beta,\gamma)\in\mathbb{R}^{2\times 2}$ by substituting $\lambda$ and $1$ for $\bm{Q}$ and $\bm{I}$ in $\bm{T}(\alpha,\beta,\gamma)$ respectively. Similar to the analysis of AFM with $\gamma=0$ in (lessard:16:aad, , Appx. A), the spectral radius of $\bm{T}_{\lambda}(\alpha,\beta,\gamma)$ is:

[TABLE]

where $r_{1}(\alpha,\beta,\gamma,\lambda)$ and $r_{2}(\alpha,\beta,\gamma,\lambda)$ denote the roots of the characteristic polynomial of $\bm{T}_{\lambda}(\cdot)$:

[TABLE]

and $\Delta(\alpha,\beta,\gamma,\lambda):=\left((1+\beta)\left(1-\alpha\lambda\right)-\gamma\alpha\lambda\right)^{2}-4\beta\left(1-\alpha\lambda\right)$ denotes the corresponding discriminant. For fixed $(\alpha,\beta,\gamma)$, the spectral radius $\rho(\bm{T}_{\lambda}(\alpha,\beta,\gamma))$ in (10) is a continuous and quasi-convex333

It is straightforward to show that $\rho(\bm{T}_{\lambda}(\alpha,\beta,\gamma))$ in (10) is quasi-convex over $\lambda$. First, $\sqrt{\beta(1-\alpha\lambda)}$ is quasi-convex over $\lambda$ (for $\Delta(\alpha,\beta,\gamma,\lambda)<0$). Second, the eigenvalue $\lambda$ satisfying $\Delta(\alpha,\beta,\gamma,\lambda)\geq 0$ is in the region where the function $\frac{1}{2}\left(|(1+\beta)\left(1-\alpha\lambda\right)-\gamma\alpha\lambda|+\sqrt{\Delta(\alpha,\beta,\gamma,\lambda)}\right)$ either monotonically increases or decreases, which overall makes the continuous function $\rho(\bm{T}_{\lambda}(\alpha,\beta,\gamma))$ quasi-convex over $\lambda$. This proof can be simply applied to other variables, i.e., $\rho(\bm{T}_{\lambda}(\alpha,\beta,\gamma))$ is quasi-convex over either $\alpha$, $\beta$ or $\gamma$.

function of $\lambda$; thus its maximum over $\lambda$ occurs at one of its boundary points $\lambda=\mu$ or $\lambda=L$.

The next section reviews the optimization of AFM coefficients to provide the fastest convergence rate, i.e., the smallest spectral radius $\rho(\bm{T}(\cdot))$ in (9), under certain constraints on $(\alpha,\beta,\gamma)$.

3.2 Review of Optimizing AFM Coefficients

under Certain Constraints on $(\alpha,\beta,\gamma)$

The AFM coefficients that provide the fastest convergence for minimizing a strongly convex quadratic function would solve

[TABLE]

Note that a heavy-ball method polyak:87 (that is not in AFM class) with similarly optimized coefficients has a linear worst-case rate with $\rho(\cdot)=\frac{1-\sqrt{q}}{1+\sqrt{q}}$ that is optimal (up to constant) for strongly convex quadratic problems nesterov:04 . Thus, optimizing (12) would be of little practical benefit for quadratic problems. Nevertheless, such optimization is new to AFM for $\gamma>0$ (with the additional constraint $\alpha={1}/{L}$ introduced below), and is useful in our later analysis for the adaptive restart in Sec. 4. A heavy-ball method with the coefficients optimized for strongly convex quadratic problems does not converge for some strongly convex nonquadratic problems lessard:16:aad , and other choices of coefficients do not yield worst-case rates that are comparable to those of some accelerated choices of AFM drori:14:pof ; lessard:16:aad , so we focus on AFM hereafter.

The coefficient optimization (12) for AFM was studied previously with various constraint. For example, optimizing (12) over $\alpha$ with the constraint $\beta=\gamma=0$ yields GM-$q$. Similarly, FGM-$q$ results from optimizing (12) over $\beta$ for the constraint444For FGM-$q$ the value of $\rho(\bm{T}_{L}({1}/{L},\beta,0))$ is [math], and the function $\rho(\bm{T}_{\mu}({1}/{L},\beta,0))$ is continuous and quasi-convex over $\beta$ (see footnote 3). The minimum of $\rho(\bm{T}_{\mu}({1}/{L},\beta,0))$ occurs at the point $\beta=\frac{1-\sqrt{q}}{1+\sqrt{q}}$ in Table 2 satisfying $\Delta\left({1}/{L},\beta,0,\mu\right)=0$, verifying the statement that FGM-$q$ results from optimizing (12) over $\beta$ given $\alpha={1}/{L}$ and $\gamma=0$.

$\alpha={1}/{L}$ and $\gamma=0$. In (lessard:16:aad, , Prop. 1), AFM with coefficients $(\alpha,\beta,\gamma)=\left(\frac{4}{\mu+3L},\frac{\sqrt{3+q}-2\sqrt{q}}{\sqrt{3+q}+2\sqrt{q}},\,0\right),$ named FGM*′*-$q$ in Table 3, was derived by optimizing (12) over $(\alpha,\beta)$ with the constraint $\gamma=0$.

Although a general unconstrained solution to (12) would be an interesting future direction, here we focus on optimizing (12) over $(\beta,\gamma)$ with the constraint $\alpha={1}/{L}$. This choice simplifies the problem (12) and is useful for analyzing an adaptive restart scheme for OGM in Sec. 4.

3.3 Optimizing the Coefficients $(\beta,\gamma)$ of AFM

When $\alpha={1}/{L}$

When $\alpha={1}/{L}$ and $\lambda=L$, the characteristic polynomial (11) becomes $r^{2}+\gamma r=0.$ The roots are $r=0$ and $r=-\gamma$, so $\rho(\bm{T}_{L}({1}/{L},\beta,\gamma))=|\gamma|$. In addition, because $\rho(\bm{T}_{\mu}({1}/{L},\beta,\gamma))$ is continuous and quasi-convex over $\beta$ (see footnote 3), it can be easily shown that the smaller value of $\beta$ satisfying the following equation:

[TABLE]

minimizes $\rho(\bm{T}_{\mu}({1}/{L},\beta,\gamma))$ for any given $\gamma$ (satisfying $\gamma\geq-1$). The optimal $\beta$ for a given $\gamma$ (when $\alpha={1}/{L}$) is

[TABLE]

which reduces to $\beta=\beta^{\star}(0)=\frac{1-\sqrt{q}}{1+\sqrt{q}}$ for FGM-$q$ (with $\gamma=0$). Substituting (14) into (10) yields $\rho(\bm{T}_{\mu}({1}/{L},\beta^{\star}(\gamma),\gamma))=|1-\sqrt{q(1+\gamma)}|,$ leading to the following simplification of (12) with $\alpha={1}/{L}$ and $\beta=\beta^{\star}(\gamma)$ from (14):

[TABLE]

The minimizer of (15) satisfies $1-\sqrt{q(1+\gamma)}=\pm\gamma,$ and with simple algebra, we get the following solutions to (12) with the constraint $\alpha={1}/{L}$ (and (15)):

[TABLE]

for which the spectral radius is $\rho^{\star}:=\rho(\bm{T}({1}/{L},\beta^{\star},\gamma^{\star}))=1-\sqrt{q(1+\gamma^{\star})}=\gamma^{\star}.$ We denote Alg. 1 with coefficients $\alpha={1}/{L}$ and ($\beta^{\star}$,$\gamma^{\star}$) in (16) as OGM-$q$.

Table 3 compares the spectral radius of the OGM-$q$ to GM-$q$, FGM-$q$, and FGM*′-$q$ (lessard:16:aad, , Prop. 1). Simple algebra shows that the spectral radius of OGM-$q$ is smaller than those of FGM-$q$ and FGM′*-$q$, i.e., $\frac{2+q-\sqrt{q^{2}+8q}}{2}\leq 1-\frac{2\sqrt{q}}{\sqrt{3+q}}\leq 1-\sqrt{q}.$ Therefore, OGM-$q$ achieves a worst-case convergence rate of $||\bm{\xi}_{k}-\bm{\xi}_{*}||^{{2}}$ that is faster than that of FGM variants (but that is slower than a heavy-ball method polyak:87 ) for a strongly convex quadratic function.

To further understand the behavior of AFM for each eigen-mode, Fig. 1 plots $\rho(\bm{T}_{\lambda}({1}/{L},\beta,\gamma))$ over $\mu\leq\lambda\leq L$ for $\mu=0.1$ and $L=1$ ($q=0.1$) as an example, where $(\beta^{\star},\gamma^{\star})=(0.4,0.6)$. The left plot of Fig. 1 first compares the $\rho(\bm{T}_{\lambda}({1}/{L},\beta,\gamma))$ values of OGM-$q$ to those of other choices of $\gamma=0,0.4,0.8$ with $\beta=\beta^{\star}(\gamma)$ in (14). The OGM-$q$ (see upper red curve in Fig. 1) has the largest value ($\rho^{\star}=\gamma^{\star}=0.6$) of $\rho(\bm{T}_{\lambda}({1}/{L},\beta,\gamma))$ at both the smallest and the largest eigenvalues ($\mu$ and $L$ respectively), unlike other choices of $\gamma$ (with $\beta^{\star}(\gamma)$) where either $\rho(\bm{T}_{\mu}({1}/{L},\beta,\allowbreak\gamma))$ or $\rho(\bm{T}_{L}({1}/{L},\beta,\gamma))$ are the largest. The other choices thus have a spectral radius $\rho(\bm{T}({1}/{L},\beta,\gamma))$ larger than that of the OGM-$q$.

The right plot of Fig. 1 illustrates $\rho(\bm{T}_{\lambda}({1}/{L},\beta,\gamma))$ values for different choices of $\beta{=0,0.2,0.4,0.6}$ for given $\gamma=\gamma^{\star}$, showing that suboptimal $\beta$ value will slow down convergence, compared to the optimal $\beta^{\star}=0.4$. AFM with $(\alpha,\beta,\gamma)=({1}/{L},0,\gamma^{\star})$ in Fig. 1 is equivalent to AFM with $\left(\frac{1}{L}(1+\gamma^{\star}),0,0\right)$, and this implies that AFM with $\beta=\gamma=0$ (e.g., GM) may have some modes for mid-valued $\lambda$ values that will converge faster than the accelerated methods, whereas its overall convergence rate (i.e., the spectral radius value) is worse. Apparently no one method can have superior convergence rates for all modes.

Similarly, although OGM-$q$ has the smallest possible spectral radius $\rho(\bm{T}(\cdot))$ among known AFM, the upper blue and red curves in the left plot of Fig. 1, corresponding to FGM-$q$ and OGM-$q$ respectively, illustrate that OGM-$q$ will have modes for large eigenvalues that converge slower than with FGM-$q$. This behavior may be undesirable when such modes of large eigenvalues dominate the overall convergence behavior.

The next section reveals that the convergence of the primary sequence $\{\bm{y}_{k}\}$ of AFM with $\alpha={1}/{L}$ is not governed by such modes of large eigenvalues unlike its secondary sequence $\{\bm{x}_{k}\}$. In addition, Fig. 1 reveals change points across $\lambda$ meaning that there are different regimes; the next section elaborates on this behavior, building upon the dynamical system analysis of AFM with $\alpha={1}/{L}$ and $\gamma=0$ in (odonoghue:15:arf, , Sec. 4).

3.4 Convergence Properties of AFM

When $\alpha={1}/{L}$

(odonoghue:15:arf, , Sec. 4) analyzed a constant-step AFM with $\alpha={1}/{L}$ and $\gamma=0$ as a linear dynamical system for minimizing a strongly convex quadratic function (3), and showed that there are three regimes of behavior for the system; low momentum, optimal momentum, and high momentum regimes. This section similarly analyzes AFM with $\alpha={1}/{L}$ and $\gamma\geq 0$ to better understand its convergence behavior when solving a strongly convex quadratic problem (3), complementing the previous section’s spectral radius analysis of AFM.

We use the eigen-decomposition of $\bm{Q}=\bm{V}\bm{\Lambda}\bm{V}^{\top}$ with $\bm{\Lambda}:=\mathop{{}\mathsf{diag}\!\left\{\lambda_{i}\right\}}\nolimits$, where the eigenvalues $\{\lambda_{i}\}$ are in an ascending order, i.e., $\mu=\lambda_{1}\leq\lambda_{2}\leq\cdots\leq\lambda_{d}=L$. And for simplicity, we let $\bm{p}=\bm{0}$ without loss of generality, leading to $\bm{x}_{*}=\bm{0}$. By defining $\bm{w}_{k}:=(w_{k,1},\cdots,w_{k,d})^{\top}=\bm{V}^{\top}\bm{y}_{k}\in\mathbb{R}^{d}$ and $\bm{v}_{k}:=(v_{k,1},\cdots,v_{k,d})^{\top}=\bm{V}^{\top}\bm{x}_{k}\in\mathbb{R}^{d}$ as the mode coefficients of the primary and secondary sequences respectively and using (4), we have the following $d$ independently evolving identical recurrence relations for the evolution of $w_{\cdot,i}$ and $v_{\cdot,i}$ of the constant-step AFM with $\alpha={1}/{L}$ respectively:

[TABLE]

for $i=1,\ldots,d$, although the initial conditions differ as follows:

[TABLE]

with $w_{0,i}=v_{0,i}$. The convergence behavior of the $i$th mode of the dynamical system of both $w_{\cdot,i}$ and $v_{\cdot,i}$ in (17) is determined by the characteristic polynomial (11) with $\alpha={1}/{L}$ and $\lambda=\lambda_{i}$. Unlike the previous sections that studied only the worst-case convergence performance using the largest absolute value of the roots of the polynomial (11), we next discuss the convergence behavior of AFM more comprehensively using (11) with $\alpha={1}/{L}$ and $\lambda=\lambda_{i}$ for the two cases

1. $\lambda_{i}=L$ and 2) $\lambda_{i}<L$.

2. $\lambda_{i}=L$: The characteristic polynomial (11) of the mode of $\lambda_{i}=L$ reduces to $r^{2}+\gamma r=0$ with two roots [math] and $-\gamma$ regardless of the choice of $\beta$. Thus we have monotone convergence for this ($d$th) mode of the dynamical system (chiang:84:fmo, , Sec. 17.1):

[TABLE]

where $c_{d}$ and $\hat{c}_{d}$ are constants depending on the initial conditions (18). Substituting $w_{1,d}=0$ and $v_{1,d}=-(\beta+\gamma)v_{0,d}$ (18) into (17) yields

[TABLE]

illustrating that the primary sequence $\{w_{k,d}\}$ reaches its optimum after one iteration, whereas the secondary sequence $\{v_{k,d}\}$ has slow monotone convergence of the distance to the optimum, while exhibiting undesirable oscillation due to the term $(-\gamma)^{k}$, corresponding to overshooting over the optimum.

2. $\lambda_{i}<L$: In (16) we found the optimal overall $\beta^{\star}$ for AFM when $\alpha={1}/{L}$. One can alternatively explore what the best value of $\beta$ would be for any given mode of the system for comparison. The polynomial (11) has repeated roots for the following $\beta$, corresponding to the smaller zero of the discriminant $\Delta({1}/{L},\beta,\gamma,\lambda_{i})$ for given $\gamma$ and $\lambda_{i}$:

[TABLE]

This choice satisfies $\beta^{\star}=\beta^{\star}(\gamma^{\star})=\beta_{1}^{\star}(\gamma^{\star})$ (16), because $\lambda_{1}$ is the smallest eigenvalue. Next we examine the convergence behavior of AFM with $\alpha={1}/{L}$ and $\gamma\geq 0$ in the following three regimes, similar to AFM with $\alpha={1}/{L}$ and $\gamma=0$ in (odonoghue:15:arf, , Sec. 4.3):555 For simplicity in the momentum analysis, we considered values $\beta$ within $[0\;1]$, containing the standard $\beta_{k}$ values in Tables 1 and 2.

This restriction excludes the effect of the $\beta$ that corresponds to the larger zero of the discriminant $\Delta({1}/{L},\beta,\gamma,\lambda_{i})$ for given $\gamma$ and $\lambda_{i}$, and that is larger than $1$. Any $\beta$ greater than $1$ has $\rho(\bm{T}_{\lambda_{i}}({1}/{L},\allowbreak\beta,\gamma))$ values (in (10) with $\alpha={1}/{L}$) that are larger than those for $\beta\in[\beta_{i}^{\star}(\gamma)\;1]$ due to the quasi-convexity of $\rho(\bm{T}_{\lambda_{i}}({1}/{L},\allowbreak\beta,\gamma))$ over $\beta$.

• •

$\beta<\beta_{i}^{\star}(\gamma)$: low momentum, over-damped,

• •

$\beta=\beta_{i}^{\star}(\gamma)$: optimal momentum, critically damped,

• •

$\beta>\beta_{i}^{\star}(\gamma)$: high momentum, under-damped.

If $\beta\leq\beta_{i}^{\star}(\gamma)$, the polynomial (11) has two real roots, $r_{1,i}$ and $r_{2,i}$ where we omit $({1}/{L},\beta,\gamma,\lambda_{i})$ in $r_{\cdot,i}=r_{\cdot}({1}/{L},\beta,\gamma,\lambda_{i})$ for simplicity. Then, the system evolves as (chiang:84:fmo, , Sec. 17.1):

[TABLE]

where constants $c_{1,i}$, $c_{2,i}$, $\hat{c}_{1,i}$ and $\hat{c}_{2,i}$ depend on the initial conditions (18). In particular, when $\beta=\beta_{i}^{\star}(\gamma)$ (21), we have the repeated root:

[TABLE]

corresponding to critical damping, yielding the fastest monotone convergence among (22) for any $\beta$ s.t.  $\beta\leq\beta_{i}^{\star}(\gamma)$. This property is due to the quasi-convexity of $\rho(\bm{T}_{\lambda_{i}}({1}/{L},\allowbreak\beta,\gamma))$ over $\beta$. If $\beta<\beta_{i}^{\star}(\gamma)$, the system is over-damped, which corresponds to the low momentum regime, where the system is dominated by the larger root that is greater than $r_{i}^{\star}(\gamma)$ (23), and thus has slow monotone convergence. However, depending on the initial conditions (18), the system may only be dominated by the smaller root, as noticed for the case $\lambda_{i}=L$ in (19) and (20). Also note that the mode of $\lambda_{i}=L$ is always in the low momentum regime regardless of the value of $\beta$.

If $\beta>\beta_{i}^{\star}(\gamma)$, the system is under-damped, which corresponds to the high momentum regime. This means that the system evolves as (chiang:84:fmo, , Sec. 17.1):

[TABLE]

where the frequency of the oscillation is given by

[TABLE]

and $c_{i}$, $\delta_{i}$, $\hat{c}_{i}$ and $\hat{\delta}_{i}$ denote constants that depend on the initial conditions (18); in particular for $\beta\approx 1$, we have $\delta_{i}\approx 0$ and $\hat{\delta}_{i}\approx 0$ so we will ignore them.

Based on the above momentum analysis, we categorize the behavior of the $i$th mode of AFM for each $\lambda_{i}$ in Fig. 1. Regimes with two curves and one curve (over $\lambda$) in Fig. 1 correspond to the low- and high-momentum regimes, respectively. In particular, for $\beta=\beta^{\star}(\gamma)$ in the left plot of Fig. 1, most $\lambda_{i}$ values (satisfying $\beta>\beta_{i}^{\star}(\gamma)$) experience high momentum (and the optimal momentum for $\lambda_{i}$ satisfying $\beta^{\star}(\gamma)=\beta_{i}^{\star}(\gamma)$, e.g., $\lambda_{i}=\mu$), whereas modes where $\lambda_{i}\approx L$ experience low momentum. The fast convergence of the primary sequence $\{w_{k,d}\}$ in (19) and (20) generalizes to the case $\lambda_{i}\approx L$, corresponding to the lower curves in Fig. 1. In addition, for $\beta{=0,0.2}$ that are smaller than $\beta^{\star}(\gamma)$ in the right plot of Fig. 1, both $\lambda\approx\mu$ and $\lambda\approx L$ experience low momentum so increasing $\beta$ improves the convergence rate.

Based on the quadratic analysis in this section, we would like to use appropriately large $\beta$ and $\gamma$ coefficients, namely $(\beta^{\star},\gamma^{\star})$, to have fast monotone convergence (for the dominating modes). However, such values require knowing the function parameter $q=\mu/L$ that is usually unavailable in practice. Using OGM (and OGM*′*) in Table 1 without knowing $q$ will likely lead to oscillation due to the high momentum (or under-damping) for strongly convex functions. The next section describes restarting schemes inspired by odonoghue:15:arf that we suggest to use with OGM to avoid such oscillation and thus heuristically accelerate the rate of OGM for a strongly convex quadratic function and even for a convex function that is locally strongly convex.

4 Restarting Schemes

Restarting an algorithm (i.e., starting the algorithm again by using the current iterate as the new starting point) after a certain number of iterations or when some restarting condition is satisfied has been found useful, e.g., for the conjugate gradient method powell:77:rpf ; nocedal:06 , called “fixed restart” and “adaptive restart” respectively. The fixed restart approach was also studied for accelerated gradient schemes such as FGM in nesterov:13:gmf ; nemirovski:94:emi . Recently adaptive restart of FGM was shown to provide dramatic practical acceleration without requiring knowledge of function parameters odonoghue:15:arf ; giselsson:14:mar ; su:16:ade . Building upon those ideas, this section reviews and applies restarting approaches for OGM. A quadratic analysis in odonoghue:15:arf justified using a restarting condition for FGM; this section extends that analysis to OGM by studying an observable quantity of oscillation that serves as an indicator for restarting the momentum of OGM.

4.1 Fixed Restart

Restarting an algorithm every $k$ iterations can yield a linear rate for decreasing a function in $\mathcal{F}_{\mu,L}(\mathbb{R}^{d})$ (nesterov:13:gmf, , Sec. 5.1) (nemirovski:94:emi, , Sec. 11.4). Suppose one restarts OGM every $k$ (inner) iterations by initializing the $(j+1)$th outer iteration using $\bm{x}_{j+1,0}=\bm{x}_{j,k}$, where $\bm{x}_{j,i}$ denotes an iterate at the $j$th outer iteration and $i$th inner iteration. Combining the OGM rate in Table 1 and the strong convexity inequality (2) yields the following linear rate for each outer iteration of OGM with fixed restart:

[TABLE]

This rate is faster than the ${4L}/{\mu k^{2}}$ rate of one outer iteration of FGM with fixed restart (using the FGM rate in Table 1). For a given $N=jk$ total number of steps, a simple calculation shows that the optimal restarting interval $k$ minimizing the rate $\left({2L}/{(\mu k^{2})}\right)^{j}$ after $N$ steps (owing from (26)) is $k_{\mathrm{fixed}}:=e\sqrt{{2}/{q}}$ that does not depend on $N$, where $e$ is Euler’s number.

There are two drawbacks of the fixed restart approach (odonoghue:15:arf, , Sec. 3.1). First, computing the optimal interval $k_{\mathrm{fixed}}$ requires knowledge of $q$ that is usually unavailable in practice. Second, using a global parameter $q$ may be too conservative when the iterates enter locally strongly convex region. Therefore, adaptive restarting odonoghue:15:arf is more useful in practice, which we review next and then apply to OGM. The above two drawbacks also apply to the methods in Table 3 that assume knowledge of the global parameter $q$.

4.2 Adaptive Restart

To circumvent the drawbacks of fixed restart, odonoghue:15:arf proposes the following two adaptive restart schemes for FGM:

• •

Function scheme for restarting (FR): restart whenever

[TABLE]

• •

Gradient scheme for restarting (GR): restart whenever

[TABLE]

These schemes heuristically improve convergence rates of FGM and both performed similarly well odonoghue:15:arf ; su:16:ade . Although the function scheme guarantees monotonic decreasing function values, the gradient scheme has two advantages over the function scheme odonoghue:15:arf ; the gradient scheme involves only arithmetic operations with already computed quantities, and it is numerically more stable.

These two schemes encourage an algorithm to restart whenever the iterates take a “bad” direction, i.e., when the function value increases or the negative gradient and the momentum have an obtuse angle, respectively. However, a convergence proof that justifies their empirical acceleration is yet unknown, so odonoghue:15:arf analyzes such restarting schemes for strongly convex quadratic functions. An alternative scheme in su:16:ade that restarts whenever the magnitude of the momentum decreases, i.e., $||\bm{y}_{k+1}-\bm{y}_{k}||<||\bm{y}_{k}-\bm{y}_{k-1}||$, has a theoretical convergence analysis for the function class $\mathcal{F}_{\mu,L}(\mathbb{R}^{d})$. However, empirically both the function and gradient schemes performed better in su:16:ade . Thus, this paper focuses on adapting practical restart schemes to OGM and extending the analysis in odonoghue:15:arf to OGM. First we introduce a new additional adaptive scheme designed specifically for AFM with $\alpha={1}/{L}$ and $\gamma>0$ (e.g., OGM).

4.3 Adaptive Decrease of $\gamma$ for

AFM with $\alpha={1}/{L}$ and $\gamma>0$

Sec. 3.4 described that the secondary sequence $\{\bm{x}_{k}\}$ of AFM with $\alpha={1}/{L}$ and $\gamma>0$ (e.g., OGM) might experience overshoot and thus slow convergence, unlike its primary sequence $\{\bm{y}_{k}\}$, when the iterates enter a region where the mode of the largest eigenvalue dominates. (Sec. 6.1.2 illustrates such an example.) From (19), the overshoot of $\bm{x}_{k}$ has magnitude proportional to $|\gamma|$, yet a suitably large $\gamma$, such as $\gamma^{\star}$ (15), is essential for overall acceleration.

To avoid (or reduce) such overshooting, we suggest the following adaptive scheme:

• •

Gradient scheme for decreasing $\gamma$ (GD$\gamma$): decrease $\gamma$ whenever

[TABLE]

Because the primary sequence $\{\bm{y}_{k}\}$ of AFM with $\alpha={1}/{L}$ is unlikely to overshoot, one could choose to simply use the primary sequence $\{\bm{y}_{k}\}$ as algorithm output instead of the secondary sequence $\{\bm{x}_{k}\}$. However, if one needs to use the secondary sequence of AFM with $\alpha={1}/{L}$ and $\gamma>0$ (e.g., Sec. 5.2), adaptive scheme (29) can help.

4.4 Observable AFM Quantities When $\alpha={1}/{L}$

This section revisits Sec. 3.4 that suggested that observing the evolution of the mode coefficients $\{w_{k,i}\}$ and $\{v_{k,i}\}$ can help identify the momentum regime. However, in practice that evolution is unobservable because the optimum $\bm{x}_{*}$ is unknown, whereas Sec. 3.4 assumed $\bm{x}_{*}=\bm{0}$. Instead we can observe the evolution of the function values, which are related to the mode coefficients as follows:

[TABLE]

and also the inner products of the gradient and momentum, i.e.,

[TABLE]

These quantities appear in the conditions for the adaptive schemes (27), (28), and (29).

One would like to increase $\beta$ and $\gamma$ as much as possible for acceleration up to $\beta^{\star}$ and $\gamma^{\star}$ (16). However, without knowing $q$ (and $\beta^{\star}$,$\gamma^{\star}$), using large $\beta$ and $\gamma$ could end up placing the majority of the modes in the high momentum regime, eventually leading to slow convergence with oscillation as described in Sec. 3.4. To avoid such oscillation, we hope to detect it using (30) and (31) and restart the algorithm. We also hope to detect the overshoot (19) of the modes of the large eigenvalues (in the low momentum regime) using (32) so that we can then decrease $\gamma$ and avoid such overshoot.

The rest of this section focuses on the case where $\beta>\beta_{1}(\gamma)$ for given $\gamma$, when the most of the modes are in the high momentum regime. Because the maximum of $\rho(\bm{T}_{\lambda}({1}/{L},\beta,\gamma))$ occurs at the points $\lambda=\mu$ or $\lambda=L$, we expect that (30), (31), and (32) will be quickly dominated by the mode of the smallest or the largest eigenvalues. Specifically, plugging $w_{k,i}$ and $v_{k,i}$ in (19), (20) and (24) to (30), (31), and (32) for only the (dominating) mode of the smallest and the largest eigenvalues ($\lambda_{1}=\mu$ and $\lambda_{d}=L$ respectively) leads to the following approximations:

[TABLE]

where $\psi_{1}=\psi_{1}(\beta,\gamma)$ in (25). Furthermore, it is likely that these expressions will be dominated by the mode of either the smallest or largest eigenvalues, so we next analyze each case separately.

4.4.1 Case 1: the Mode of the Smallest Eigenvalue Dominates

When the mode of the smallest eigenvalue dominates, we further approximate (33) as

[TABLE]

using simple trigonometric identities and the approximations $\sqrt{\beta(1-{\mu}/{L})}\approx 1$ and $\mathop{{}{\operator@font sin}\!\left(k\psi_{1}\right)}\nolimits\approx\mathop{{}{\operator@font sin}\!\left((k+{1}/{2}\right)}\nolimits\psi_{1})$ for small $\mu$ (leading to small $\psi_{1}$ in (25)). The values (34) exhibit oscillations at a frequency proportional to $\psi_{1}(\beta,\gamma)$ in (25). This oscillation can be detected by the conditions (27) and (28) and is useful in detecting the high momentum regime where a restart can help improve the convergence rate.

4.4.2 Case 2: the Mode of the Largest Eigenvalue Dominates

Unlike the primary sequence $\{\bm{y}_{k}\}$ of AFM with $\alpha={1}/{L}$ (e.g., OGM), convergence of its secondary sequence $\{\bm{x}_{k}\}$ may be dominated by the mode of the largest eigenvalue in (19) and (20). By further approximating (33) for the case when the mode of the largest eigenvalue dominates, the function value $f(\bm{x}_{k})\approx\frac{1}{2}L\hat{c}_{d}^{2}\,\gamma^{2k}$ decreases slowly but monotonically, whereas $f(\bm{y}_{k})\approx f(\bm{x}_{*})=0$ and $\mathop{\langle-\nabla f(\bm{x}_{k}),\,\bm{y}_{k+1}-\bm{y}_{k}\rangle}\nolimits\approx 0.$ Therefore, neither restart condition (27) or (28) can detect such non-oscillatory observable values, even though the secondary mode $\{w_{k,d}\}$ of the largest eigenvalue is oscillating (corresponding to overshooting over the optimum). However, the inner product of two sequential gradients:

[TABLE]

can detect the overshoot of the secondary sequence $\{\bm{x}_{k}\}$, suggesting that the algorithm should adapt by decreasing $\gamma$ when condition (29) holds. Decreasing $\gamma$ too much may slow down the overall convergence rate when the mode of the smallest eigenvalue is not negligible. Thus, we use (29) only when using the secondary sequence $\{\bm{x}_{k}\}$ as algorithm output (e.g., Sec. 5.2).

5 Proposed Adaptive Schemes for OGM

5.1 Adaptive Scheme of OGM for Smooth and Strongly Convex Problems

Alg. 2 illustrates a new adaptive version of OGM*′* (rather than OGM)666

OGM requires choosing the number of iterations $N$ in advance for computing $\theta_{N}$ in Table 1, which seems incompatible with adaptive restarting schemes. In contrast, the parameters $t_{k}$ in Table 1 and Alg. 2 are independent of $N$. The fact that $\theta_{N}$ is larger than $t_{N}$ at the last ($N$th) iteration helps to dampen (by reducing the values of $\beta$ and $\gamma$) the final update to guarantee a faster (optimal) worst-case rate for the last secondary iterate $\bm{x}_{N}$. This property was studied in kim:17:otc . We could perform one last update using $\theta_{N}$ after a restart condition is satisfied, but this step appears unnecessary because restarting already has the effect of dampening (reducing $\beta$ and $\gamma$). Thus, Alg. 2 uses OGM*′* instead that uses $t_{k}$ and that has a worst-case rate that is similar to that of OGM. that is used in our numerical experiments in Sec. 6. When a restart condition is satisfied in Alg. 2, we reset $t_{k}=1$ to discard the previous momentum that has a bad direction. When the decreasing $\gamma$ condition is satisfied in Alg. 2, we decrease $\sigma$ to suppress undesirable overshoot of the secondary sequence $\{\bm{x}_{k}\}$. Although the analysis in Sec. 3 considered only strongly convex quadratic functions, the numerical experiments in Sec. 6 illustrate that the adaptive scheme is also useful more generally for smooth convex functions in $\mathcal{F}_{0,L}(\mathbb{R}^{d})$, as described in (odonoghue:15:arf, , Sec. 4.6).

5.2

Adaptive Scheme of a Proximal Version of OGM for Nonsmooth Composite Convex Problems

Modern applications often involve nonsmooth composite convex problems:

[TABLE]

where $f\in\mathcal{F}_{0,L}(\mathbb{R}^{d})$ is a smooth convex function (typically not strongly convex) and $\phi\in\mathcal{F}_{0,\infty}(\mathbb{R}^{d})$ is a convex function that is possibly nonsmooth and “proximal-friendly” combettes:11:psm , such as the $\ell_{1}$ regularizer $\phi(\bm{x})=||\bm{x}||_{1}$. Our numerical experiments in Sec. 6 show that a new adaptive version of a proximal variant of OGM can be useful for solving such problems.

To solve (36), beck:09:afi developed a fast proximal gradient method, popularized under the name fast iterative shrinkage-thresholding algorithm (FISTA). FISTA has the same rate as FGM in Table 1 for solving (36), by simply replacing the line 3 of Alg. 1 with FGM coefficients by $\bm{y}_{k+1}=\operatorname{prox}_{\alpha\phi}(\bm{x}_{k}-\alpha\nabla f(\bm{x}_{k}))$, where the proximity operator is defined as $\operatorname{prox}_{h}(\bm{z}):=\mathop{\operator@font arg\,min}_{\bm{x}}\{\frac{1}{2}||\bm{z}-\bm{x}||^{2}+h(\bm{x})\}.$ Variants of FISTA with adaptive restart were studied in (odonoghue:15:arf, , Sec. 5.2).

Inspired by the fact that OGM has a worst-case rate faster than FGM, taylor:17:ewc studied a proximal variant777 Applying the proximity operator to the primary sequence $\{\bm{y}_{k}\}$ of OGM, similar to the extension of FGM to FISTA, leads to a poor worst-case rate taylor:17:ewc . Therefore, taylor:17:ewc applied the proximity operator to the secondary sequence of OGM and showed numerically that this version has a worst-case rate about twice faster than that of FISTA. of OGM (POGM). It is natural to pursue acceleration of POGM888 Like OGM, POGM in (taylor:17:ewc, , Sec. 4.3) requires choosing the number of iterations $N$ in advance for computing $\theta_{N}$, and this is incompatible with adaptive restarting schemes. Therefore, analogous to using OGM*′* instead of OGM for an adaptive scheme in Alg. 2 (see footnote 6), Alg. 3 uses a proximal version of OGM*′* (rather than the POGM in taylor:17:ewc ) with restart. An extension of OGM*′* (without restart) to a proximal version with a fast worst-case rate is unknown yet by using variations of any (or all) of the three adaptive schemes (27), (28), (29), as illustrated in Alg. 3. Regarding a function restart condition for POGM, we use

[TABLE]

instead of $F(\bm{y}_{k+1})>F(\bm{y}_{k})$, because $F(\bm{y}_{k})$ can be unbounded (e.g., $\bm{y}_{k}$ can be unfeasible for constrained problems). For gradient conditions of POGM, we consider the composite gradient mapping $G(\bm{x}_{k})\in\nabla f(\bm{x}_{k})+\partial\phi(\bm{x}_{k+1})$ in Alg. 3 that differs from the standard composite gradient mapping in nesterov:13:gmf . We then use the gradient conditions

[TABLE]

for restarting POGM or decreasing $\gamma$ of POGM respectively. Here POGM must output the secondary sequence $\{\bm{x}_{k}\}$ because the function value $F(\bm{y}_{k})$ of the primary sequence may be unbounded. This situation was the motivation for (29) (and the second inequality of (38)) and Sec. 4.3. When $\phi(\bm{x})=0$, Alg. 3 reduces to an algorithm that is similar to Alg. 2, where only the location of the restart and decreasing $\gamma$ conditions differs.

6 Numerical Results

This section shows the results of applying OGM*′* and POGM*′* with adaptive schemes in Alg. 2 and Alg. 3 to various numerical examples including both strongly convex quadratic problems and non-strongly convex problems.999 Software for the algorithms and for producing the figures in Sec. 6 is available at https://gitlab.eecs.umich.edu/michigan-fast-optimization/ogm-adaptive-restart.

(For simplicity, we omit the prime symbol of OGM*′* and POGM*′* with adaptive restart hereafter.) The results illustrate that OGM (or POGM) with adaptive schemes converges faster than FGM (or FISTA) with adaptive restart. The plots show the decrease of $F(\bm{y}_{k})$ of the primary sequence for FGM (FISTA) and OGM unless specified. For POGM, we use the secondary sequence $\{\bm{x}_{k}\}$ as an output and plot $F(\bm{x}_{k})$, since $F(\bm{y}_{k})$ can be unbounded.

6.1 Strongly Convex Quadratic Examples

This section considers two types of strongly convex quadratic examples, where the mode of either the smallest eigenvalue or the largest eigenvalue dominates, providing examples of the analysis in Sec. 4.4.1 and 4.4.2 respectively.

6.1.1 Case 1: the Mode of the Smallest Eigenvalue Dominates

Fig. 2 compares GM, FGM and OGM, with or without the knowledge of $q$, for minimizing a strongly convex quadratic function (3) in $d=500$ dimensions with $q=10^{-4}$, where we generated $\bm{A}$ (for $\bm{Q}=\bm{A}^{\top}\bm{A}$) and $\bm{p}$ randomly. As expected, knowing $q$ accelerates convergence.

Fig. 2 also illustrates that adaptive restart helps FGM and OGM to nearly achieve the fast linear converge rate of their non-adaptive versions that know $q$. As expected, OGM variants converge faster than FGM variants for all cases. In Fig. 2, ‘FR’ and ‘GR’ stand for function restart (27) and gradient restart (28), respectively, and both behave nearly the same.

6.1.2 Case 2: the Mode of the Largest Eigenvalue Dominates

Consider the strongly convex quadratic function with \bm{Q}=\left[\begin{array}[]{cc}q&0\\ 0&1\end{array}\right], $q=0.01$, $\bm{p}=\bm{0}$ and $\bm{x}_{*}=\bm{0}$. When starting the algorithm from the initial point $\bm{x}_{0}=(0.2,\;1),$ the secondary sequence $\{\bm{x}_{k}\}$ of OGM-GR101010 Fig. 3 only compares the results of the gradient restart (GR) scheme for simplicity, where the function restart (FR) behaves similarly.

(or equivalently OGM-GR-GD$\gamma$ $(\bar{\sigma}=1.0)$) is dominated by the mode of largest eigenvalue in Fig. 3, illustrating the analysis of Sec. 4.4.2. Fig. 3 illustrates that the primary sequence of OGM-GR converges faster than that of FGM-GR, whereas the secondary sequence of OGM-GR initially converges even slower than GM. To deal with such slow convergence coming from the overshooting behavior of the mode of the largest eigenvalue of the secondary sequence of OGM, we employ the decreasing $\gamma$ scheme in (29). Fig. 3 shows that using $\bar{\sigma}<1$ in Alg. 2 leads to overall faster convergence of the secondary sequence $\{\bm{x}_{k}\}$ than the standard OGM-GR where $\bar{\sigma}=1$. We leave optimizing the choice of $\bar{\sigma}$ or studying other strategies for decreasing $\gamma$ as future work.

6.2 Non-strongly Convex Examples

This section applies adaptive OGM (or POGM) to three non-strongly convex numerical examples in odonoghue:15:arf ; su:16:ade . The numerical results show that adaptive OGM (or POGM) converges faster than FGM (or FISTA) with adaptive restart.

6.2.1 Log-Sum-Exp

The following function from odonoghue:15:arf is smooth but non-strongly convex:

[TABLE]

It approaches $\max_{i=1,\ldots,m}(\bm{a}_{i}^{\top}\bm{x}-b_{i})$ as $\eta\to 0$. Here, $\eta$ controls the function smoothness $L=\frac{1}{\eta}\lambda_{\max}(\bm{A}^{\top}\bm{A})$ where $\bm{A}=[\bm{a}_{1}\cdots\bm{a}_{m}]^{\top}\in\mathbb{R}^{m\times d}$. The region around the optimum is approximately quadratic since the function is smooth, and thus the adaptive restart can be useful without knowing the local condition number.

For $(m,d)=(100,20)$, we randomly generated $\bm{a}_{i}\in\mathbb{R}^{d}$ and $b_{i}\in\mathbb{R}$ for $i=1,\ldots,m$, and investigated $\eta=1,10$. Fig. 4 shows that OGM with adaptive restart converges faster than FGM with the adaptive restart. The benefit of adaptive restart is dramatic here; apparently FGM and OGM enter a locally strongly convex region after about $100-200$ iterations, where adaptive restart then provide a fast linear rate.

6.2.2 Sparse Linear Regression

Consider the following cost function used for sparse linear regression:

[TABLE]

for $\bm{A}\in\mathbb{R}^{m\times d}$, where $L=\lambda_{\max}(\bm{A}^{\top}\bm{A})$ and the parameter $\tau$ balances between the measurement error and signal sparsity. The proximity operator becomes a soft-thresholding operator, e.g., \operatorname{prox}_{\zeta_{k+1}\phi}(\bm{x})=\mathop{{}{\operator@font sgn}\!\left(\bm{x}\right)}\nolimits\max\big{\{}|\bm{x}|-\zeta_{k+1}\tau,0\big{\}}. The minimization seeks a sparse solution $\bm{x}_{*}$, and often the cost function is strongly convex with respect to the non-zero elements of $\bm{x}_{*}$. Thus we expect to benefit from adaptive restarting.

For each choice of $(m,d,s,\tau)$ in Fig. 5, we generated an $s$-sparse true vector $\bm{x}_{\mathrm{true}}$ by taking the $s$ largest entries of a randomly generated vector. We then simulated $\bm{b}=\bm{A}\bm{x}_{\mathrm{true}}+\bm{\varepsilon}$, where the entries of matrix $\bm{A}$ and vector $\bm{\varepsilon}$ were sampled from a zero-mean normal distribution with variances $1$ and $0.1$ respectively. Fig. 5 illustrates that POGM with adaptive schemes provide acceleration over FISTA with adaptive restart. While Sec. 3.4 discussed the undesirable overshooting behavior that a secondary sequence of OGM (or POGM) may encounter, these examples rarely encountered such behavior. Therefore the choice of $\bar{\sigma}$ in the adaptive POGM was not significant in this experiment, unlike Sec. 6.1.2.

6.2.3 Constrained Quadratic Programming

Consider the following box-constrained quadratic program:

[TABLE]

where $L=\lambda_{\max}(\bm{Q})$. The ISTA (a proximal variant of GM), FISTA and POGM use the projection operator: $\operatorname{prox}_{\frac{1}{L}\phi}(\bm{x})=\operatorname{prox}_{\zeta_{k+1}\phi}(\bm{x})=\min\{\max\{\bm{x},\bm{l}\},\bm{u}\}.$ Fig. 6 denotes each algorithm by a projected GM, a projected FGM, and a projected OGM respectively. Similar to Sec. 6.2.2, after the algorithm identifies the active constraints the problem typically becomes a strongly convex quadratic problem where we expect to benefit from adaptive restart.

Fig. 6 studies two examples with problem dimensions $d=500,1000$, where we randomly generate a positive definite matrix $\bm{Q}$ having a condition number $10^{7}$ (i.e., $q=10^{-7}$), and a vector $\bm{p}$. Vectors $\bm{l}$ and $\bm{u}$ correspond to the interval constraints $-1\leq x_{i}\leq 1$ for $\bm{x}=\{x_{i}\}$. The optimum $\bm{x}_{*}$ had $47$ and $81$ active constraints out of $500$ and $1000$ respectively. In Fig. 6, the projected OGM with adaptive schemes converged faster than FGM with adaptive restart and other non-adaptive algorithms.

7 Conclusions

We introduced adaptive restarting schemes for the optimized gradient method (OGM) to heuristically provide a fast linear convergence rate when the function is strongly convex or even when the function is not globally strongly convex. The method resets the momentum when it makes a bad direction. We provided a heuristic dynamical system analysis to justify the practical acceleration of the adaptive scheme of OGM, by extending the existing analysis of the fast gradient method (FGM). On the way, we described a new accelerated gradient method named OGM-$q$ for strongly convex quadratic problems. Numerical results illustrate that the proposed adaptive approach practically accelerates the convergence rate of OGM, and in particular, performs faster than FGM with adaptive restart. An interesting open problem is to determine the worst-case rates for OGM (and FGM) with adaptive restart.

Acknowledgements.

This research was supported in part by NIH grant U01 EB018753.