Accelerated Extra-Gradient Descent: A Novel Accelerated First-Order Method

TL;DR

This paper introduces AXGD, a novel accelerated first-order method based on a predictor-corrector approach, achieving optimal convergence rates for smooth functions and demonstrating robustness and versatility compared to traditional methods like Nesterov's AGD.

Contribution

The paper presents AXGD, a new accelerated method that differs from AGD, utilizing a predictor-corrector scheme inspired by Mirror-Prox and Extra-Gradient, with a novel primal-dual analysis and broad applicability.

Findings

Achieves asymptotically optimal convergence rates for smooth functions.

Matches Nesterov's method performance while showing increased robustness to noise.

Extends to objectives with generalized smoothness and non-smooth Lipschitz properties.

Abstract

We provide a novel accelerated first-order method that achieves the asymptotically optimal convergence rate for smooth functions in the first-order oracle model. To this day, Nesterov's Accelerated Gradient Descent (AGD) and variations thereof were the only methods achieving acceleration in this standard blackbox model. In contrast, our algorithm is significantly different from AGD, as it relies on a predictor-corrector approach similar to that used by Mirror-Prox [Nemirovski, 2004] and Extra-Gradient Descent [Korpelevich, 1977] in the solution of convex-concave saddle point problems. For this reason, we dub our algorithm Accelerated Extra-Gradient Descent (AXGD). Its construction is motivated by the discretization of an accelerated continuous-time dynamics [Krichene et al., 2015] using the classical method of implicit Euler discretization. Our analysis explicitly shows the effects of discretization through a conceptually novel primal-dual viewpoint. Moreover, we show that the method is quite general: it attains optimal convergence rates for other classes of objectives (e.g., those with generalized smoothness properties or that are non-smooth and Lipschitz-continuous) using the appropriate choices of step lengths. Finally, we present experiments showing that our algorithm matches the performance of Nesterov's method, while appearing more robust to noise in some cases.

Equations

$$\forall x,y\in X,\Vert \nabla f(x)-\nabla f(y){\Vert }_{\ast }\le L\cdot \Vert x-y\Vert .$$

$$\nabla {\psi }^{\ast }(\mathbf{z})=\arg \underset{\mathbf{x}\in X}{\max }\left\{⟨\mathbf{z},\mathbf{x}⟩-\psi (\mathbf{x})\right\}.$$

$$\begin{array}{c}{\mathbf{x}}^{(k+1)}=\frac{A_k}{A_{k+1}}{\hat{\mathbf{x}}}^{(k)}+\frac{a_{k+1}}{A_{k+1}}\nabla {\psi }^{\ast }({\mathbf{z}}^{(k)}),\\ {\mathbf{z}}^{(k+1)}={\mathbf{z}}^{(k)}-a_{k+1}\nabla f({\mathbf{x}}^{(k+1)}),\\ {\hat{\mathbf{x}}}^{(k+1)}=\mathrm{Grad}({\mathbf{x}}^{(k+1)}).\end{array}$$

$$\begin{array}{c}{\hat{\mathbf{x}}}^{(k)}=\frac{A_k}{A_{k+1}}{\mathbf{x}}^{(k)}+\frac{a_{k+1}}{A_{k+1}}\nabla {\psi }^{\ast }({\mathbf{z}}^{(k)}),\\ {\hat{\mathbf{z}}}^{(k)}={\mathbf{z}}^{(k)}-a_{k+1}\nabla f({\hat{\mathbf{x}}}^{(k)}),\\ {\mathbf{x}}^{(k+1)}=\frac{A_k}{A_{k+1}}{\mathbf{x}}^{(k)}+\frac{a_{k+1}}{A_{k+1}}\nabla {\psi }^{\ast }({\hat{\mathbf{z}}}^{(k)}),\\ {\mathbf{z}}^{(k+1)}={\mathbf{z}}^{(k)}-a_{k+1}\nabla f({\mathbf{x}}^{(k+1)}).\end{array}$$

$$f(\mathbf{u})\ge \frac{{\int }_{t_0}^{t}f({\mathbf{x}}^{(\tau )})d{\alpha }^{(\tau )}}{A^{(t)}}+\frac{{\int }_{t_0}^t⟨\nabla f({\mathbf{x}}^{(\tau )}),\mathbf{u}-{\mathbf{x}}^{(\tau )}⟩d{\alpha }^{(\tau )}}{A^{(t)}},\forall \mathbf{u}\in X.$$

$$f({\mathbf{x}}^{\ast })+\frac{D_{\psi }({\mathbf{x}}^{\ast },{\mathbf{x}}^{(t_0)})}{A^{(t)}}$$

$$\ge \frac{{\int }_{t_0}^{t}f({\mathbf{x}}^{(\tau )})d{\alpha }^{(\tau )}}{A^{(t)}}+\frac{{\min }_{\mathbf{u}\in X}\left\{{\int }_{t_0}^t⟨\nabla f({\mathbf{x}}^{(\tau )}),\mathbf{u}-{\mathbf{x}}^{(\tau )}⟩d{\alpha }^{(\tau )}+D_{\psi }(\mathbf{u},{\mathbf{x}}^{(t_0)})\right\}}{A^{(t)}}.$$

$$L_t\stackrel{\mathrm{def}}{=}$$

$$+\frac{({\alpha }^{(t)}-A^{(t)})f({\mathbf{x}}^{\ast })-D_{\psi }({\mathbf{x}}^{\ast },{\mathbf{x}}^{(t_0)})}{{\alpha }^{(t)}}.$$

$$\nabla {\psi }^{\ast }({\mathbf{z}}^{(t)})=\arg \underset{\mathbf{u}\in X}{\min }\left\{{\int }_{t_0}^t⟨\nabla f({\mathbf{x}}^{(\tau )}),\mathbf{u}-{\mathbf{x}}^{(\tau )}⟩d{\alpha }^{(\tau )}+D_{\psi }(\mathbf{u},{\mathbf{x}}^{(t_0)})\right\}.$$

$$\frac{d}{dt}({\alpha }^{(t)}{G}_t)$$

$$=⟨\nabla f({\mathbf{x}}^{(t)}),{\alpha }^{(t)}{\mathbf{x}}^{(t)}-{\dot{\alpha }}^{(t)}\left(\nabla {\psi }^{\ast }({\mathbf{z}}^{(t)})-{\mathbf{x}}^{(t)}\right)⟩.$$

$$\begin{array}{c}{\dot{\mathbf{z}}}^{(t)}=-{\dot{\alpha }}^{(t)}\nabla f({\mathbf{x}}^{(t)}),\\ {\dot{\mathbf{x}}}^{(t)}={\dot{\alpha }}^{(t)}\frac{\nabla {\psi }^{\ast }({\mathbf{z}}^{(t)})-{\mathbf{x}}^{(t)}}{{\alpha }^{(t)}},\\ {\mathbf{z}}^{(t_0)}=\nabla \psi ({\mathbf{x}}^{(t_0)}),{\mathbf{x}}^{(t_0)}\in X\text{ is an arbitrary initial point.}\end{array}$$

$$f({\mathbf{x}}^{(t)})-f({\mathbf{x}}^{\ast })\le \frac{{\alpha }^{(t_0)}(f({\mathbf{x}}^{(t_0)})-f({\mathbf{x}}^{\ast }))+D_{\psi }({\mathbf{x}}^{\ast },{\mathbf{x}}^{(t_0)})}{{\alpha }^{(t)}}.$$

$$G_k=\frac{A_1}{A_k}{G}_1+\frac{{\sum }_{i=1}^k{E}_i}{A_k}$$

$$E_{i+1}\le ⟨\nabla f({\mathbf{x}}^{(i+1)}),A^{(i+1)}{\mathbf{x}}^{(i+1)}-A^{(i)}{\mathbf{x}}^{(i)}-a_{i+1}\nabla {\psi }^{\ast }({\mathbf{z}}^{(i+1)})⟩-D_{{\psi }^{\ast }}({\mathbf{z}}^{(i)},{\mathbf{z}}^{(i+1)}).$$

$$I^{(i,i+1)}=a_{i+1}⟨\nabla f({\mathbf{x}}^{(i+1)}),\nabla {\psi }^{\ast }({\mathbf{z}}^{(i+1)})-{\mathbf{x}}^{(i+1)}⟩.$$

$$I_c^{(i,i+1)}$$

$$=A^{(i)}(f({\mathbf{x}}^{(i+1)})-f({\mathbf{x}}^{(i)}))-{\int }_i^{i+1}⟨{\dot{\mathbf{z}}}^{(\tau )},\nabla {\psi }^{\ast }({\mathbf{z}}^{(i+1)})-\nabla {\psi }^{\ast }({\mathbf{z}}^{(\tau )})⟩d\tau$$

$$=A^{(i)}(f({\mathbf{x}}^{(i+1)})-f({\mathbf{x}}^{(i)}))-D_{{\psi }^{\ast }}({\mathbf{z}}^{(i)},{\mathbf{z}}^{(i+1)}),$$

$$E_{i+1}\le ⟨\nabla f({\mathbf{x}}^{(i+1)}),A^{(i+1)}{\mathbf{x}}^{(i+1)}-A^{(i)}{\mathbf{x}}^{(i)}-a_{i+1}\nabla {\psi }^{\ast }({\mathbf{z}}^{(i+1)})⟩-D_{{\psi }^{\ast }}({\mathbf{z}}^{(i)},{\mathbf{z}}^{(i+1)}),$$

$$\{\begin{array}{l}{\mathbf{z}}^{(k+1)}={\mathbf{z}}^{(k)}-a_{k+1}\nabla f({\mathbf{x}}^{(k+1)}),\\ {\mathbf{x}}^{(k+1)}=\frac{A_k}{A_{k+1}}{\mathbf{x}}^{(k)}+\frac{a_{k+1}}{A_{k+1}}\nabla {\psi }^{\ast }({\mathbf{z}}^{(k+1)})\end{array}$$

$${\mathbf{x}}^{(k+1)}=\frac{A_k}{A_{k+1}}{\mathbf{x}}^{(k)}+\frac{a_{k+1}}{A_{k+1}}\nabla {\psi }^{\ast }({\mathbf{z}}^{(k)}-a_{k+1}\nabla f({\mathbf{x}}^{(k+1)})).$$

$$\{\begin{array}{l}{\hat{\mathbf{x}}}^{(k)}=\frac{A_k}{A_{k+1}}{\mathbf{x}}^{(k)}+\frac{a_{k+1}}{A_{k+1}}\nabla {\psi }^{\ast }({\mathbf{z}}^{(k)}).\\ {\mathbf{x}}^{(k+1)}=\frac{A_k}{A_{k+1}}{\mathbf{x}}^{(k)}+\frac{a_{k+1}}{A_{k+1}}\nabla {\psi }^{\ast }({\mathbf{z}}^{(k)}-a_{k+1}\nabla f({\hat{\mathbf{x}}}^{(k)}))\end{array}$$

$$E_{i+1}\le$$

$$-D_{{\psi }^{\ast }}({\hat{\mathbf{z}}}^{(i)},{\mathbf{z}}^{(i+1)})-D_{{\psi }^{\ast }}({\mathbf{z}}^{(i)},{\hat{\mathbf{z}}}^{(i)}).$$

$$E_{i+1}\le$$

$$=$$

$$-D_{{\psi }^{\ast }}({\mathbf{z}}^{(i)},{\mathbf{z}}^{(i+1)}).$$

$$a_{i+1}⟨\nabla f({\hat{\mathbf{x}}}^{(i)}),\nabla {\psi }^{\ast }({\hat{\mathbf{z}}}^{(i)})-\nabla {\psi }^{\ast }({\mathbf{z}}^{(i+1)})⟩=⟨{\mathbf{z}}^{(i)}-{\hat{\mathbf{z}}}^{(i)},\nabla {\psi }^{\ast }({\hat{\mathbf{z}}}^{(i)})-\nabla {\psi }^{\ast }({\mathbf{z}}^{(i+1)})⟩$$

$$=D_{{\psi }^{\ast }}({\mathbf{z}}^{(i)},{\mathbf{z}}^{(i+1)})-D_{{\psi }^{\ast }}({\hat{\mathbf{z}}}^{(i)},{\mathbf{z}}^{(i+1)})-D_{{\psi }^{\ast }}({\mathbf{z}}^{(i)},{\hat{\mathbf{z}}}^{(i)}),$$

$$f({\mathbf{x}}^{(k)})-f({\mathbf{x}}^{\ast })\le \frac{D_{\psi }({\mathbf{x}}^{\ast },{\hat{\mathbf{x}}}^{(0)})}{A_k}.$$

$$f({\mathbf{x}}^{(k)})-f({\mathbf{x}}^{\ast })\le \frac{2L}{(k+1{)}^2}\Vert {\mathbf{x}}^{\ast }-{\hat{\mathbf{x}}}^{(0)}{\Vert }^2.$$

$$⟨\nabla f({\mathbf{x}}^{(k+1)})-\nabla f({\hat{\mathbf{x}}}^{(k)}),\nabla {\psi }^{\ast }({\hat{\mathbf{z}}}^{(k)})-\nabla {\psi }^{\ast }({\mathbf{z}}^{(k+1)})⟩$$

$$\le L\Vert {\mathbf{x}}^{(k+1)}-{\hat{\mathbf{x}}}^{(k)}\Vert \cdot \Vert \nabla {\psi }^{\ast }({\mathbf{z}}^{(k+1)})-\nabla {\psi }^{\ast }({\hat{\mathbf{z}}}^{(k)})\Vert ,$$

Full text

Its construction is motivated by the discretization of an accelerated continuous-time dynamics [15] using the classical method of implicit Euler discretization. Our analysis explicitly shows the effects of discretization through a conceptually novel primal-dual viewpoint. Moreover, we show that the method is quite general: it attains optimal convergence rates for other classes of objectives (e.g., those with generalized smoothness properties or that are non-smooth and Lipschitz-continuous) using the appropriate choices of step lengths. Finally, we present experiments showing that our algorithm matches the performance of Nesterov’s method, while appearing more robust to noise in some cases.

1 Introduction

First-order methods for convex optimization have come to play an important role in the design of algorithms and in Theoretical Computer Science in general, with applications including numerical methods [30, 13], graph algorithms [12, 29], submodular optimization [8] and complexity theory [11].

A classical setting for convex optimization is that of smooth optimization, i.e., minimizing a convex differentiable function $f$ over a convex set $X\subseteq\mathbb{R}^{n}$, with the smoothness assumption that the gradient of $f$ be $L$-Lipschitz continuous111Lipschitz continuity is defined w.r.t to a pair of dual norms $\|\cdot\|,\|\cdot\|_{*}$. At a first reading, these can be taken as $\|\cdot\|_{2}.$ for some positive real $L,$ i.e.:

[TABLE]

In this setting, it is also assumed that the algorithm can access the input function $f$ only via queries to a first-order oracle, i.e., a blackbox that on input $x\in X$, returns the vector $\nabla f(x)$ in constant time.222In general, we may assume that the blackbox also returns the function value $f(x)$. However, for the general class of problems we consider this information is not necessary and the gradient suffices [28]. For intuition about this, see the expression for the change in duality gap in Equation 2.2.

Smooth optimization is of particular interest because it is the simplest setting in which the phenomenon of acceleration arises, i.e., the optimal algorithms in the blackbox model achieve an error that scales as $O(1/t^{2}),$ where $t$ is the number of queries [22]. This should be compared to the convergence of steepest-descent methods, which attempt to locally minimize the first-order approximation to the function and only yield $O(1/t)$-convergence [3, 28]. Acceleration has proved an active topic of algorithmic research, both for the promise of obtaining generic speed-ups for problems having some smoothness condition and for the unintuitive nature of the fact that faster algorithms can be obtained by not moving in the direction of steepest-descent.

Recently, a number of papers have helped demystify the concept behind accelerated algorithms by providing interpretations based on continuous dynamics and their discretization [15, 33, 31], geometric ideas [5], and on trading off the performances of two slower first-order methods [1]. Despite these efforts, to this day, Nesterov’s Accelerated Gradient Descent (agd) methods remain the only paradigm [22, 23] through which to obtain accelerated algorithms in the blackbox model and in related settings, where all existing accelerated algorithms are variations of Nesterov’s general method [32].

Our Main Contributions

We present a novel accelerated first-order method that achieves the optimal convergence rate for smooth functions and is significantly different from Nesterov’s method, as it relies on a predictor-corrector approach, similar to that of Mirror-Prox [18] and Extra-Gradient Descent [14]. For this reason, we name our method Accelerated Extra-Gradient Descent (axgd). Our derivation of the axgd algorithm is based on the discretization of a recently proposed continuous-time accelerated algorithm [15, 33]. The continuous-time view is particularly helpful in clarifying the relation between agd, axgd, and Mirror-Prox. Following [15], given a gradient field $\nabla f$ and a prox function $\psi$, it is possible to define two continuous-time evolutions: the mirror-descent dynamics and the accelerated-mirror-descent dynamics (see Section 2.2). With this setup, Nesterov’s agd can be seen as a variant of the classical forward-Euler discretization applied to the accelerated-mirror-descent dynamics. In contrast, Mirror-Prox and extra-gradient methods arise from an approximate backward-Euler discretization [9] on the mirror-descent dynamics. Finally, our algorithm axgd is the result of an approximate backward-Euler discretization of the accelerated mirror-descent dynamics.

Another conceptual contribution of our paper is the application of a primal-dual viewpoint on the convergence of first-order methods, both in continuous and discrete time. At every time instant $t$, our algorithm explicitly maintains a current primal solution $\mathbf{x}^{(t)}$ and a current dual solution $\mathbf{z}^{(t)}$, the latter in the form of a convex combination of gradients of the convex objective, i.e., a lower-bounding hyperplane. This primal-dual pair of solutions yields, for every $t,$ both an upper bound $U_{t}$ and a lower bound $L_{t}$ on the optimum: $U_{t}\geq f(\mathbf{x}^{*})\geq L_{t}.$ In all cases, we obtain convergence bounds by explicitly quantifying the rate at which the duality gap $G_{t}=U_{t}-L_{t}$ goes to zero. We believe that this primal-dual viewpoint makes the analysis and design of first-order methods easier to carry out. We provide its application to proving other classical results in first-order methods, including Mirror Descent, Mirror-Prox, and Frank-Wolfe algorithms in the upcoming manuscript [7].

Other Technical Contributions

In Section 2.6, we provide a unified convergence proof for standard smooth functions (as defined above) and for functions with Hölder-continuous gradients, a more general notion of smoothness [20]. While this paper focuses on the standard smooth setup, the same techniques easily yield results matching those of agd methods for the strongly-convex-and-smooth case. Indeed, it is possible to prove that our method is universal, in the sense of Nesterov [21], meaning that it can be composed with a line-search algorithm to yield near-optimal algorithms even when the smoothness parameters of the functions are unknown. We illustrate this phenomenon by showing that (axgd) also achieves the optimal rate for the optimization of Lipschitz-continuous convex functions, a non-smooth problem.

Finally, we present a suite of experiments comparing agd, axgd, and standard gradient methods, showing that the performance of axgd closely matches that of agd methods. We also explore the empirical performance of axgd in the practically and theoretically relevant case in which the queried gradients are corrupted by noise. We show that axgd exhibits better stability properties than agd in some cases, leading to a number of interesting theoretical questions on the convergence of axgd.

1.1 Related Work

In his seminal work [22, 23], Nesterov gave a method for the minimization of convex functions that are smooth with respect to the Euclidean norm, where the function is accessed through a first-order oracle. Nesterov’s method converges quadratically faster than gradient descent, at a rate of $O(\frac{1}{t^{2}})$, which has been shown to be asymptotically optimal [23] for smooth functions in this standard blackbox model [28]. More recently, Nesterov generalized this method to allow non-Euclidean norms in the definition of smoothness [25]. We refer to this generalization of Nesterov’s method and to instantiations thereof as agd methods. Accelerated gradient methods have been widely extended and modified for different settings, including composite optimization [27, 16], cubic regularization [26], and universal methods [21]. They have also found a number of fundamental applications in many algorithmic areas, including machine learning (see [4]) and discrete optimization [17].

An important application of agd  methods concerns the solution of various convex-concave saddle point problems. While these are examples of non-smooth problems, for which the optimal rate is known to be $\Omega(\frac{1}{\sqrt{k}})$ [20], Nesterov showed that the saddle-point structure can be exploited by smoothing the original problem and applying agd methods on the resulting smooth function [25]. This approach [25, 24] yields an $O(\frac{1}{k})$-convergence for convex-concave saddle point problems with smooth gradients. Surprisingly, at around the same time, Nemirovski [18] gave a very different algorithm, known as Mirror-Prox, which achieves the same complexity for the saddle point problem. Mirror-Prox does not rely on the algorithm or analysis underlying agd, but is based instead on the idea of an extra-gradient step, i.e., a correction step that is performed at every iteration to speed up convergence. Mirror-Prox can be viewed as an approximate solution to the implicit Euler discretization of the standard mirror descent dynamics of Nemirovski and Yudin [20]. In this fashion, our axgd algorithm resembles Mirror-Prox as it also makes use of an approximate implicit Euler step to discretize a different, accelerated dynamic.

A number of interpretations have been proposed to explain the phenomenon of acceleration in first-order methods. Tseng gives a formal framework that unifies all the different instantiations of agd methods [32]. More recently, Allen-Zhu and Orecchia [1] cast agd methods as the result of coupling mirror descent and gradient descent steps. Bubeck et al. give an elegant geometric interpretation of the Euclidean instantiation of Nesterov’s method [5]. At the same time, Su et al. [31], Krichene et al. [15], and Wibisono et al. [33] have provided characterizations of accelerated methods as discretizations of certain families of ODEs related to the gradient flow of the objective $f.$ Our algorithm is strongly influenced by these works: in particular, the starting point for the derivation of axgd is the continuous-time accelerated-mirror-descent (amd) dynamics [15].

1.2 Preliminaries

We focus on continuous and differentiable functions defined on a closed convex set $X\subseteq\mathbb{R}^{n}$. We assume that there is an arbitrary (but fixed) norm $\|\cdot\|$ associated with the space, and all the statements about function properties are stated with respect to that norm. We also define the dual norm $\|\cdot\|_{*}$ in the standard way: $\|\mathbf{z}\|_{*}=\sup\{{\left\langle\mathbf{z},\mathbf{x}\right\rangle}:\|\mathbf{x}\|=1\}$. The following definitions will be useful in our analysis, and thus we state them here for completeness.

Definition 1.1**.**

A function $f:X\rightarrow\mathbb{R}$ is convex on $X$, if for all $\mathbf{x},\mathbf{\hat{x}}\in X$: $f(\mathbf{\hat{x}})\geq f(\mathbf{x})+\left\langle\nabla f(\mathbf{x}),\mathbf{\hat{x}}-\mathbf{x}\right\rangle$.

Definition 1.2**.**

A function $f:X\rightarrow\mathbb{R}$ is smooth on $X$ with smoothness parameter $L$ and with respect to a norm $\|\cdot\|$, if for all $\mathbf{x},\mathbf{\hat{x}}\in X$: $f(\mathbf{\hat{x}})\leq f(\mathbf{x})+\left\langle\nabla f(\mathbf{x}),\mathbf{\hat{x}}-\mathbf{x}\right\rangle+\frac{L}{2}\|\mathbf{\hat{x}}-\mathbf{x}\|^{2}$.

Definition 1.2 can equivalently be stated as: $\|\nabla f(\mathbf{x})-\nabla f(\mathbf{\hat{x}})\|_{*}\leq L\|\mathbf{x}-\mathbf{\hat{x}}\|$.

Definition 1.3**.**

A function $f:X\rightarrow\mathbb{R}$ is strongly convex on $X$ with strong convexity parameter $\sigma$ and with respect to a norm $\|\cdot\|$, if for all $\mathbf{x},\mathbf{\hat{x}}\in X$: $f(\mathbf{\hat{x}})\geq f(\mathbf{x})+\left\langle\nabla f(\mathbf{x}),\mathbf{\hat{x}}-\mathbf{x}\right\rangle+\frac{\sigma}{2}\|\mathbf{\hat{x}}-\mathbf{x}\|^{2}$.

Definition 1.4**.**

(Bregman Divergence) $D_{\psi}(\mathbf{x},\mathbf{\hat{x}})\stackrel{{\scriptstyle\mathrm{\scriptscriptstyle def}}}{{=}}\psi(\mathbf{x})-\psi(\mathbf{\hat{x}})-\left\langle\nabla\psi(\mathbf{\hat{x}}),\mathbf{x}-\mathbf{\hat{x}}\right\rangle$.

Definition 1.5**.**

(Convex Conjugate) Function $\psi^{*}$ is the convex conjugate of $\psi:X\rightarrow\mathbb{R}$, if $\psi^{*}(\mathbf{z})=\max_{\mathbf{x}\in X}\{\left\langle\mathbf{z},\mathbf{x}\right\rangle-\psi(\mathbf{x})\}$, $\forall\mathbf{z}\in\mathbb{R}$.

In the rest of the paper, we will assume that $\psi(\mathbf{x})$ is continuously differentiable, so that Fenchel-Moreau Theorem implies that $\psi^{**}=\psi$.333Note that Fenchel-Moreau Theorem requires $\psi$ to only be lower-semicontinuous for $\psi^{**}=\psi$ to hold, which is a weaker property than continuity or continuous differentiability. We are interested in minimizing a convex function $f$ over $X\subseteq\mathbb{R}^{n}$. We let $\mathbf{x}^{*}=\arg\min_{\mathbf{x}\in X}f(\mathbf{x})$.

We will refer to any step that decreases the value of $f$ as a gradient descent step. In the special case of a smooth function $f$ the gradient descent step from a point $\mathbf{x}\in X$ will be given as $\mathrm{Grad}(\mathbf{x})=\arg\min_{\mathbf{\hat{x}}\in X}\{f(\mathbf{x})+\left\langle\nabla f(\mathbf{x}),\mathbf{\hat{x}}-\mathbf{x}\right\rangle+\frac{L}{2}\|\mathbf{\hat{x}}-\mathbf{x}\|^{2}\}$.

We will assume that there is a strongly-convex differentiable function $\psi:X\rightarrow\mathbb{R}$ such that $\max_{\mathbf{x}\in X}\{\left\langle\mathbf{z},\mathbf{x}\right\rangle-\psi(\mathbf{x})\}$ is easily solvable, possibly in a closed form. Notice that this problem defines the convex conjugate of $\psi(\cdot)$, i.e., $\psi^{*}(\mathbf{z})=\max_{\mathbf{x}\in X}\{\left\langle\mathbf{z},\mathbf{x}\right\rangle-\psi(\mathbf{x})\}$. The following standard fact will be extremely useful in carrying out the analysis of the algorithms in this paper.

Fact 1.6**.**

Let $\psi:X\to\mathbb{R}$ be a differentiable strongly-convex function. Then:

[TABLE]

Additional useful properties of Bregman divergence are provided in Appendix A.

2 Accelerated Extra-Gradient Descent

In this section, we describe the axgd method and analyze its convergence. For comparison, steps of agd and axgd are shown next to each other in the box below. In continuous time, both algorithms follow the same dynamics. However, due to the different discretization methods used in constructing agd and axgd, they follow different discrete-time updates. In particular, we show in [7] that agd can be interpreted as performing explicit (forward) Euler discretization plus a gradient step to correct the discretization error. In contrast, axgd uses an approximate implementation of implicit (backward) Euler discretization to directly control the discretization error.

**Accelerated Gradient Descent (agd)

**

$\begin{gathered}\mathbf{x}^{(k+1)}=\frac{A_{k}}{A_{k+1}}\mathbf{\hat{x}}^{(k)}+\frac{a_{k+1}}{A_{k+1}}\nabla\psi^{*}(\mathbf{z}^{(k)}),\\ \mathbf{z}^{(k+1)}=\mathbf{z}^{(k)}-a_{k+1}\nabla f(\mathbf{x}^{(k+1)}),\\ \mathbf{\hat{x}}^{(k+1)}=\mathrm{Grad}(\mathbf{x}^{(k+1)}).\end{gathered}$

(2.1)

Accelerated Extra-Gradient Descent (axgd)

$\begin{gathered}\mathbf{\hat{x}}^{(k)}=\frac{A_{k}}{A_{k+1}}\mathbf{x}^{(k)}+\frac{a_{k+1}}{A_{k+1}}\nabla\psi^{*}(\mathbf{z}^{(k)}),\\ \mathbf{\hat{z}}^{(k)}=\mathbf{z}^{(k)}-a_{k+1}\nabla f(\mathbf{\hat{x}}^{(k)}),\\ \mathbf{x}^{(k+1)}=\frac{A_{k}}{A_{k+1}}\mathbf{x}^{(k)}+\frac{a_{k+1}}{A_{k+1}}\nabla\psi^{*}(\mathbf{\hat{z}}^{(k)}),\\ \mathbf{z}^{(k+1)}=\mathbf{z}^{(k)}-a_{k+1}\nabla f(\mathbf{x}^{(k+1)}).\end{gathered}$

(2.2)

The idea behind axgd is similar to the dual-averaging version of Nemirovski’s mirror prox algorithm [18, 7], with the main difference coming from the discretization of the accelerated dynamics in Equation (2.5) (as opposed to the standard mirror descent dynamics used in [18]). As we will show, an exact implicit Euler step would have $\nabla\psi^{*}(\mathbf{z}^{(k+1)})$ instead of $\nabla\psi^{*}(\mathbf{\hat{z}}^{(k)})$ in the third line of axgd. However, obtaining $\mathbf{x}^{(k+1)}$ in a such a manner could be computationally prohibitive since $\mathbf{z}^{(k+1)}$ implicitly depends on $\mathbf{x}^{(k+1)}$ through its gradient. Instead, we opt for an extra prox-step $\nabla\psi^{*}(\mathbf{\hat{z}}^{(k)})$ that adds the gradient at an intermediate point $\mathbf{\hat{x}}^{(k)}$ constructed using $\mathbf{x}^{(k)}$ and $\mathbf{z}^{(k)}$ from the previous iteration. Thanks to this extra-gradient step, axgd can correct the discretization error without using a gradient step.

Convergence proof for axgd together with the sufficient conditions for obtaining optimal convergence bounds are provided in Section 2.4. For example, Theorem 2.5 shows that when the objective function is smooth, axgd converges at the optimal rate of $1/k^{2}$. The analysis of agd is provided in [7].

2.1 Approximate Optimality Gap

The analysis relies on the construction of an approximate optimality gap $G_{t}$, which is defined as the difference of an upper bound $U_{t}$ and a lower bound $L_{t}$ to the optimal function value $f(\mathbf{x}^{*})$. In particular, for an increasing function of time $t$, $\alpha^{(t)}$, the convergence analysis will work on establishing the following:

Invariance condition: $\alpha^{(t)}G_{t}$ is non-increasing with time $t$.

Such a condition immediately implies: $G_{t}\leq\frac{\alpha^{(t_{0})}}{\alpha^{(t)}}G_{t_{0}}$, leading to the $\frac{1}{\alpha^{(t)}}$ convergence rate. We sketch the main ideas that relate to the accelerated methods and axgd in particular here for completeness, while the more general arguments that recover a number of known first-order methods are provided in [7].

We now describe the upper bound and the lower bound choices, which will take the same form in both continuous time and discrete time domains. To do so, we will rely on the Lebesgue-Stieltjes integration, which allows us to treat continuous and discrete choice of $\alpha^{(t)}$ in a unified manner. Observe that when $\alpha^{(t)}$ is a discrete measure, $\dot{\alpha}^{(t)}$ is a train of (scaled) Dirac Delta functions. Denote $A^{(t)}=\int_{t_{0}}^{t}d\alpha^{(\tau)}=\int_{t_{0}}^{t}\dot{\alpha}^{(\tau)}d\tau$.

Upper Bound

As $\mathbf{x}^{*}\in X$ is the minimizer of $f(\cdot)$, $f(\mathbf{x})$ for any $\mathbf{x}\in X$ constitutes a valid upper bound. In particular, our choice of the upper bound will be $U_{t}=f(\mathbf{x}^{(t)})$, where $\mathbf{x}^{(t)}$ is the solution maintained by the algorithm at time $t$.

Lower Bound

More interesting than the upper bound is the construction of a lower bound to $f(\mathbf{x}^{*})$. From convexity of $f$, we have the standard lower-bounding hyperplanes $\forall\mathbf{x},\mathbf{\hat{x}}\in X$: $f(\mathbf{x})\geq f(\mathbf{\hat{x}})+\left\langle\nabla f(\mathbf{\hat{x}}),\mathbf{x}-\mathbf{\hat{x}}\right\rangle$. A natural choice of a lower bound to the optimum at time $t\geq t_{0},$ is obtained by averaging such hyperplanes over $[t_{0},t]$ according to the measure $\alpha$:

[TABLE]

While we could take the minimum over $\mathbf{u}\in X$ on the right-hand side of this equation as our notion of lower bound, this choice has two serious drawbacks. First, it is non-smooth, and in general not even differentiable, as a function of $t.$ Second, in continuous-time, it is not defined for our initial time $t_{0},$ meaning that we do not have a natural concept of initial lower bound and initial duality gap. (In the discrete time, we can ensure that $\alpha$ contains a Dirac Delta function at $t_{0}$, which overcomes this issue.) We address the first problem by applying regularization, i.e., by adding to both sides of the inequality a regularizer term that is strongly-convex in $\mathbf{x}$ and then minimizing the right-hand side with respect to $\mathbf{u}\in X$.444This is similar to the well-known Moreau-Yosida regularization. Without loss of generality, the regularizer can be taken to be the Bregman divergence of a $\sigma$-strongly convex function $\psi$ taken from an input point $\mathbf{x}^{(t_{0}).}$ This yields:

[TABLE]

To address the second problem, we mix into the $\alpha$-combination of hyperplanes the optimal lower bound $f(\mathbf{x}^{*})$ with weight $\alpha^{(t)}-A^{(t)}$ (which is just zero in the discrete time, as in that case $A^{(t)}=\alpha^{(t)}$). Rescaling the normalization factor, we obtain our notion of regularized lower bound:

[TABLE]

2.2 Accelerated Mirror Descent in Continuous Time

We now show that the accelerated dynamics can be obtained by enforcing the invariance condition from previous subsection with $\alpha^{(t)}G_{t}$ being constant; i.e., we enforce that $\frac{d}{dt}(\alpha^{(t)}G_{t})=0$. Towards that goal, assume that $\alpha^{(t)}$ is continuously differentiable, and observe that $\alpha^{(t)}-A^{(t)}=\alpha^{(t_{0})}$ is constant. To simplify the notation when taking the time derivative of $\alpha^{(t)}G^{(t)}$, we first show the following:

Proposition 2.1**.**

Let $\mathbf{z}^{(t)}=\nabla\psi(\mathbf{x}^{(t_{0})})-\int_{t_{0}}^{t}\nabla f(\mathbf{x}^{(\tau)})d\alpha^{(\tau)}$. Then:

[TABLE]

I.e., $\nabla\psi^{*}(\mathbf{z}^{(t)})$ is the argument of the minimum appearing in the definition of lower bound $L_{t}$. The proof is simple and is provided in the appendix.

Recalling that $U_{t}=f(\mathbf{x}^{(t)})$ and using (2.3) and Danskin’s theorem (which allows us to differentiate inside the min):

[TABLE]

Hence, to obtain $\frac{d}{dt}(\alpha^{(t)}G_{t})=0$, it suffices to set $\alpha^{(t)}\mathbf{x}^{(t)}=\dot{\alpha}^{(t)}(\nabla\psi^{*}(\mathbf{z}^{(t)})-\mathbf{x}^{(t)})$, resulting in the accelerated dynamics from [15]:

[TABLE]

It is not hard to see that (2.5) constructs a sequence of points $\mathbf{x}^{(t)}$ that are feasible, that is, $\mathbf{x}^{(t)}\in X$. This is because $\mathbf{x}^{(t)}$ can equivalently be written as $\frac{d}{dt}(\alpha^{(t)}\mathbf{x}^{(t)})=\dot{\alpha}^{(t)}\nabla\psi^{*}(\mathbf{z}^{(t)})$, which, after integrating over $\tau\in[t_{0},t]$, gives $\mathbf{x}^{(t)}=\frac{\alpha^{(t_{0})}}{\alpha^{(t)}}\mathbf{x}^{(t_{0})}+\frac{1}{\alpha^{(t)}}\int_{t_{0}}^{t}\nabla\psi^{*}(\mathbf{z}^{(\tau)})d\alpha^{(\tau)}$ – a convex combination of $\mathbf{x}^{(t_{0})}$ and $\nabla\psi^{*}(\mathbf{z}^{(\tau)})$ for $\tau\in[t_{0},t]$. By (2.5), $\mathbf{x}^{(t_{0})}\in X$, while $\nabla\psi^{*}(\mathbf{z}^{(\tau)})\in X$ by Proposition 2.1.

We immediately obtain the following continuous-time convergence guarantee:

Lemma 2.2**.**

Let $\mathbf{x}^{(t)}$ evolve according to (2.5). Then, $\forall t\geq t_{0}$:

[TABLE]

Proof.

We have already established that $\frac{d}{dt}(\alpha^{(t)}G^{(t)})=0$, and, therefore, $f(\mathbf{x}^{(t)})-f(\mathbf{x}^{*})\leq G_{t}=\frac{\alpha^{(t_{0})}}{\alpha^{(t)}}G_{t_{0}}$. Observing that $G_{t_{0}}=f(\mathbf{x}^{(t_{0})})-f(\mathbf{x}^{*})+D_{\psi}(\mathbf{x}^{*},\mathbf{x}^{(t_{0})})/\alpha^{(t_{0})}$, the proof follows. ∎

2.3 Discretization

As discussed in Section 2.1, our construction of the approximate optimality gap is valid both in the continuous time and in the discrete time domain. To understand where the discretization error occurs, we make the following observations. First, the upper bound does not involve any integration, and thus cannot incur a discretization error. In the lower bound (2.3), the role of the first integral is only to perform weighted averaging, which is the same in the continuous time and in the discrete time, and, therefore, does not incur a discretization error. The terms that are not integrated over look the same whether or not $\alpha^{(t)}$ is discrete. Therefore, the only term that can incur the discretization error is the integral under the min: $I^{(t_{0},t)}=\int_{t_{0}}^{t}\left\langle\nabla f(\mathbf{x}^{(\tau)}),\nabla\psi^{*}(\mathbf{z}^{(t)})-\mathbf{x}^{(\tau)}\right\rangle d\alpha^{(\tau)}$.

As mentioned before, when $\alpha$ is a discrete measure, we can express it as $\alpha^{(t)}=\sum_{i=1}^{\infty}a_{i}\delta(t-(t_{0}+i-1))$, where $\delta(\cdot)$ denotes the Dirac Delta function and $a_{i}$’s are positive. Then $A^{(t)}=\int_{t_{0}}^{t}d\alpha^{(\tau)}=\sum_{i:t_{0}+i-1\leq t}a_{i}$. To simplify the notation, we will use $i\in\mathbb{Z}_{+}$ to denote the discrete time points corresponding to $t_{0}+i-1$ on the continuous line. Therefore, the discretization error incurred in $A^{(t)}L_{t}$ between the discrete time points $i$ and $i+1$ (understood as integrating from $i^{+}$ to $(i+1)^{+}$) is $I^{(i,i+1)}-I_{c}^{(i,i+1)}$, where $I_{c}^{(i,i+1)}$ is the continuous approximation of $I^{(i,i+1)}$ (i.e., we allow continuous integration rules in $I_{c}^{(i,i+1)}$). We can now establish the following bound on the discretization error.

Lemma 2.3**.**

Let $A_{i+1}G_{i+1}-A_{i}G_{i}\equiv E_{i+1}$ be the discretization error. Then

[TABLE]

and

[TABLE]

Proof.

The first part of the lemma follows by summing over $1\leq i\leq k$. For the second part, we have already argued that $E_{i+1}=I_{c}^{(i,i+1)}-I^{(i,i+1)}$. For the discrete integral $I^{(i,i+1)}$, as $\dot{\alpha}^{(t)}$ just samples the function under the integral at point $i+1$, we have:

[TABLE]

For the continuous integral, using (2.5) and integration by parts:

[TABLE]

where we have used $\dot{\mathbf{z}}^{(\tau)}=-\dot{\alpha}^{(\tau)}\nabla f(\mathbf{x}^{(\tau)})$, $\nabla_{\mathbf{z}^{(\tau)}}D_{\psi^{*}}(\mathbf{z}^{(\tau)},\mathbf{z}^{(i+1)})=\nabla\psi^{*}(\mathbf{z}^{(\tau)})-\nabla\psi^{*}(\mathbf{z}^{(i+1)})$, and $D_{\psi^{*}}(\mathbf{z}^{(i)},\mathbf{z}^{(i)})=0$.

By convexity of $f$, $f(\mathbf{x}^{(i+1)})-f(\mathbf{x}^{(i)})\leq\left\langle\nabla f(\mathbf{x}^{(i+1)}),\mathbf{x}^{(i+1)}-\mathbf{x}^{(i)}\right\rangle$. Combining with (2.6) and (2.7):

[TABLE]

as claimed. ∎

We remark that the same result for the discretization error can be obtained by directly computing $A_{i+1}G_{i+1}-A_{i}G_{i}$ under a discrete measure $\alpha$ (where all the integrals in the definition of the duality gap are replaced by summations). We have chosen to work with the integration error described above to demonstrate the cause of the discretization error.

We now describe how axgd cancels out the discretization error by (approximately) implementing implicit Euler discretization of $\dot{\mathbf{x}}^{(t)}$.

Implicit Euler Discretization

Implicit Euler discretization is an abstract discretization method which defines the next iterate $\mathbf{x}^{(k+1)}$ implicitly as a function of the gradient at $\mathbf{x}^{(k+1)}$. In the case of the amd dynamics, implicit Euler discretization yields the following algorithm: let $\mathbf{x}^{(1)}\in X$ be an arbitrary initial point that satisfies $\mathbf{x}^{(1)}=\nabla\psi^{*}(\mathbf{z}^{(1)})$, where $\mathbf{z}^{(1)}=\nabla\psi(\mathbf{x}^{(1)})-\nabla f(\mathbf{x}^{(1)})$; for all $k\geq 1$

[TABLE]

Observe that $\mathbf{x}^{(k+1)}$ in (2.8) exactly sets the inner product in $E_{i+1}$ (Lemma 2.3) to zero, leaving only the negative term $-D_{\psi^{*}}(\mathbf{z}^{(i)},\mathbf{z}^{(i+1)})$. While this discretization is not computationally feasible in practice, as it requires solving for the implicitly defined $\mathbf{x}^{(k+1)},$ it also boasts a negative discretization error, i.e., it converges faster than the continuous-time amd. Ultimately, we will use this extra slack to trade-off the error arising from an approximate implicit discretization.

2.4 Convergence of AXGD

A standard way to implement implicit Euler discretization in the solution of ODEs [9] is to replace the exact solution of the implicit equation with a small number of fixed point iterations of the same equation. In our case, the implicit equation can be written as:

[TABLE]

Two steps of the fixed-point iteration yield the following updates, which are exactly those performed by axgd:

[TABLE]

We can now analyze axgd as producing an approximate solution to the implicit Euler discretization problem. The following lemma gives a general bound on the convergence of axgd for a convex and differentiable $f(\cdot)$ without additional assumptions. The only (mild) difference is replacing $D_{\psi}(\mathbf{x},\mathbf{x}^{(1)})$ and $D_{\psi}(\mathbf{x}^{*},\mathbf{x}^{(1)})$ by $D_{\psi}(\mathbf{x},\mathbf{\hat{x}}^{(0)})$ and $D_{\psi}(\mathbf{x}^{*},\mathbf{\hat{x}}^{(0)})$, since we start from the “intermediate” point $\mathbf{\hat{x}}^{(0)}$. This change is only important for bounding the initial gap $G_{1}$; everything else is the same as before.

Lemma 2.4**.**

Consider the axgd  algorithm as described in Equation (2.2), starting from an arbitrary point $\mathbf{\hat{x}}^{(0)}$ with $\mathbf{z}^{(0)}=\nabla\psi(\mathbf{\hat{x}}^{(0)})$ and $A_{0}=0$. Then the error from Lemma 2.3 is bounded by:

[TABLE]

Proof.

From Lemma 2.3:

[TABLE]

We now use the fact that $a_{i+1}f(\mathbf{\hat{x}}^{(i)})=\mathbf{z}^{(i)}-\mathbf{\hat{z}}^{(i)}$ together with the standard triangle-inequality for Bregman divergences (see Proposition A.3) to show that:

[TABLE]

Combining the results of the last two equations, we get the claimed bound on the error. ∎

2.5 Smooth Minimization with AXGD

We show that axgd achieves the asymptotically optimal convergence rate of $1/k^{2}$ for the minimization of an $L$-smooth convex objective $f(\cdot)$ by applying Lemma 2.4. The crux of the proof is that we can take sufficiently large steps while keeping the error from Lemma 2.4 non-positive. In other words, we are able to move quickly through the continuous evolution of amd by taking large discrete steps.

Theorem 2.5**.**

Let $f:X\rightarrow\mathbb{R}$ be an $L$-smooth convex function and let $\mathbf{x}^{(k)},\mathbf{z}^{(k)},\mathbf{\hat{x}}^{(k)},\mathbf{\hat{z}}^{(k)}$ be updated according to the axgd algorithm in Equation (2.2), starting from an arbitrary initial point $\mathbf{\hat{x}}^{(0)}\in X$ with the following initial conditions: $\mathbf{z}^{(0)}=\nabla\psi(\mathbf{\hat{x}}^{(0)})$ and $A_{0}=0$. Let $\psi:X\rightarrow\mathbb{R}$ be $\sigma$-strongly convex. If $\frac{{a_{k}}^{2}}{A_{k}}\leq\frac{\sigma}{L},$ then for all $k\geq 1$,

[TABLE]

In particular, if $a_{k}=\frac{k+1}{2}\cdot\frac{\sigma}{L}$ and $\psi(\mathbf{x})=\frac{\sigma}{2}\|\mathbf{x}\|^{2}$, then:

[TABLE]

Proof.

The proof follows directly by applying Lemma 2.4 and using $L$-smoothness of $f$ and $\sigma$-strong convexity of $\psi$. In particular, by Cauchy-Schwartz inequality and smoothness:

[TABLE]

and, by Proposition A.2

[TABLE]

From the definition of the steps, $\mathbf{x}^{(k+1)}-\mathbf{x}^{(k)}=\frac{a_{k+1}}{A_{k+1}}(\nabla\psi^{*}(\mathbf{\hat{z}}^{(k)})-\nabla\psi^{*}(\mathbf{z}^{(k)}))$, and, therefore:

[TABLE]

where $p=\|\nabla\psi^{*}(\mathbf{\hat{z}}^{(k)})-\nabla\psi^{*}(\mathbf{z}^{(k+1)})\|$ and $q=\|\nabla\psi^{*}(\mathbf{z}^{(k)})-\nabla\psi^{*}(\mathbf{\hat{z}}^{(k)})\|$. Since, for any $p,q$, $p^{2}+q^{2}-2\alpha pq\geq 0$ whenever $\alpha\leq 1$, it follows that $E_{k+1}\leq 0$ whenever $\frac{{a_{k+1}}^{2}}{A_{k+1}}\frac{L}{\sigma}\leq 1$, which is true by the theorem assumptions. In particular, for $a_{k}=\frac{k+1}{2}\cdot\frac{\sigma}{L}$, $A_{k}=\frac{\sigma}{L}(\frac{(k+1)(k+2)}{4})\geq\frac{\sigma}{L}\frac{(k+1)^{2}}{4}$. This proves that $f(\mathbf{x}^{(k)})-f(\mathbf{x}^{*})\leq\frac{G_{1}}{A_{k}}.$ It remains to bound $G_{1}.$ This a simple computation, shown in the appendix, which yields: $G_{1}\leq\frac{1}{A_{1}}D_{\psi}(\mathbf{x}^{*},\mathbf{\hat{x}}^{(0)}).$ ∎

2.6 Generalized Smoothness: Hölder-Continuous Gradients

Suppose that $f(\cdot)$ has Hölder-continuous gradients, namely, $f(\cdot)$ then satisfies:

[TABLE]

which also implies:

[TABLE]

where $\nu\in(0,1]$, $L_{\nu}\in\mathbb{R}_{++}$. In particular, if $\nu=1$, then $f(\cdot)$ is $L_{\nu}$-smooth. Thus, the functions with Hölder-continuous gradients represent a class of functions with generalized/relaxed smoothness properties.

The lower iteration complexity bound for (unconstrained) minimization of convex functions with Hölder-continuous gradients was established in [20] and equals $O\left({L_{\nu}D_{1}^{1+\nu}}{\epsilon^{-\frac{2}{1+3\nu}}}\right)$, where $D_{1}$ is the distance from the initial point to the optimal solution. A matching upper bound was obtained in [19].

To recover the optimal convergence rate in the minimization of convex functions with Hölder-continuous gradients, as before, we bound the discretization error from Lemma 2.4. Before doing so, we will need the following technical proposition (which appears in a similar form as Lemma 3.1 a) in [18]).

Proposition 2.6**.**

[TABLE]

The proof is provided in the appendix.

Theorem 2.7**.**

Let $f(\cdot)$ be a convex function that satisfies (2.10), and let $\psi(\cdot)$ be $\sigma$-strongly convex. Let $\mathbf{x}^{(k)},\mathbf{z}^{(k)},\mathbf{\hat{x}}^{(k)},\mathbf{\hat{z}}^{(k)}$ be updated according to the axgd algorithm in Equation (2.2), starting from an arbitrary initial point $\mathbf{\hat{x}}^{(0)}\in X$ with the following initial conditions: $\mathbf{z}^{(0)}=\nabla\psi(\mathbf{\hat{x}}^{(0)})$ and $A_{0}=0$. Let $a_{k}=c\frac{\sigma}{L_{\nu}}D^{1-\nu}k^{\frac{-1+3\nu}{2}}$, where $D=\max_{\mathbf{x},\mathbf{\hat{x}}\in X}\|\mathbf{x}-\mathbf{\hat{x}}\|$ and $c=2^{\frac{3\nu(\nu+1)-1}{2}}$. Then, $\forall k\geq 1:$

[TABLE]

In particular, if $\psi(\mathbf{x})=\frac{\sigma}{2}\|\mathbf{x}\|^{2}$, then:

[TABLE]

Proof.

We prove the theorem by bounding the discretization error $E_{i+1}$ from Lemma 2.4. Applying Propositions A.2 and 2.6:

[TABLE]

where the second inequality is by (2.10) and the third inequality is by the step definition (2.2).

Taking $a_{k}=c\frac{\sigma}{L_{\nu}}D^{1-\nu}k^{\frac{-1+3\nu}{2}}$, where $c=2^{\frac{3\nu(\nu+1)-1}{2}}$, it follows that A_{k}=\sum_{i=1}^{k}a_{i}\geq\sum_{i=\lceil k/2\rceil}^{k}a_{i}\geq\frac{c}{2}D^{1-\nu}\frac{\sigma}{L_{\nu}}\big{(}\frac{k}{2}\big{)}^{\frac{1+3\nu}{2}}. Therefore, the expression in (2.12) is at least:

[TABLE]

as $(k+1)^{\nu-1}\leq 1$. Therefore, we have that $G_{k}\leq\frac{A^{(1)}}{A^{(k)}}G_{1}$, and using similar arguments to bound the initial gap $G_{1}$, the proof follows. ∎

2.7 Non-Smooth Minimization: Lipschitz-Continuous Objective

We now show that we can recover the well-known $\frac{1}{\sqrt{k}}$ convergence rate for the class of non-smooth $L$-Lipschitz objectives by using axgd. This is summarized in the following theorem. We note that, as in the analysis of classical mirror descent (see, e.g., [3]), the factor $\log(k)$ can be removed if we fix the approximation error (and, consequently, the number of steps $k$) in advance.

Theorem 2.8**.**

Let $f(\cdot)$ be a Lipschitz-continuous function with parameter $L$. Let $\mathbf{x}^{(k)}$, $\mathbf{z}^{(k)}$, $\mathbf{\hat{x}}^{(k)}$, $\mathbf{\hat{z}}^{(k)}$ be updated according to the axgd algorithm in Equation (2.2), starting from an arbitrary initial point $\mathbf{\hat{x}}^{(0)}\in X$ with the following initial conditions: $\mathbf{z}^{(0)}=\nabla\psi(\mathbf{\hat{x}}^{(0)})$ and $A_{0}=0$. If $a_{k}=\frac{\sqrt{\sigma}}{2\sqrt{2}L}\sqrt{\frac{D_{\psi}(\mathbf{x}^{*},\mathbf{\hat{x}}^{(0)})}{k}}$, then, $\forall k\geq 1$:

[TABLE]

In particular, for $\psi(\mathbf{x})=\frac{\sigma}{2}\|\mathbf{x}\|^{2}$:

[TABLE]

Proof.

As before, we bound the discretization error from Lemma 2.4. As $f(\cdot)$ is $L$-Lipschitz, using Proposition A.2:

[TABLE]

where we have used the inequality $2xy-x^{2}\leq y^{2}$, $\forall x,y$. As $\sigma\geq L$ and

[TABLE]

we have that

[TABLE]

which, after bounding the initial gap by similar arguments, completes the proof. ∎

3 Experiments

We now illustrate the performance of agd and axgd for (i) an unconstrained problem over $\mathbb{R}^{n}$ with the objective function $f(\mathbf{x})=\frac{1}{2}\left\langle\mathbf{A}\mathbf{x},\mathbf{x}\right\rangle-\left\langle\mathbf{b},\mathbf{x}\right\rangle$, and (ii) for the problem with the same objective and unit simplex as the feasible region, where $\mathbf{A}$ is the Laplacian of a cycle graph555Namely, the sum of a tridiagonal matrix $\mathbf{B}$ with 2’s on its main diagonal and -1’s on its remaining two diagonals and a matrix $\mathbf{C}$ whose all elements are zero except for the $\mathbf{C}_{1,n}=\mathbf{C}_{n,1}=-1$. and $\mathbf{b}$ is a vector whose first element is one and the remaining elements are zero. This example is known as a “hard” instance for smooth minimization – it is typically used in proving the lower iteration complexity bound for first-order methods (see, e.g., [28]). We also include Gradient Descent (gd) in the exact gap graphs for comparison. In the experiments, we take $n=100$ and $\sigma=L$ ($=4$). We use the $\ell_{2}$ norm in the gradient steps.

In the figures, $f$ denotes the objective value at the upper-bound point and $f^{*}$ denotes the optimal objective value, so that $f-f^{*}$ is the true distance to the optimum (the exact gap). Fig. 1 shows the distance to the optimum and the approximate duality gap $G_{k}=U_{k}-L_{k}$ obtained using our analysis. We can observe that agd and axgd exhibit similar performance in these examples. The approximate gap overestimates the actual duality gap, however, the difference between the two decreases with the number of iterations.

Acceleration and Noise

We now consider the setting in which the gradients output by our oracle are corrupted by additive noise, which has significant applications in practice [10] and theory [2]. We note that this model is fundamentally different from the inexact model considered by Devolder et al. [6], for which tight lower bounds preventing acceleration exist.666In [6], it is assumed that a function $f(\cdot)$ is associated with a $(\delta,L)$ oracle, such that $f(\mathbf{\hat{x}})\leq f(\mathbf{x})+\left\langle\nabla f(\mathbf{x}),\mathbf{\hat{x}}-\mathbf{x}\right\rangle+\frac{L}{2}\|\mathbf{\hat{x}}-\mathbf{x}\|^{2}+\frac{\delta}{2},\;\forall\mathbf{x},\mathbf{\hat{x}}\in X.$ Such a model seems more suitable for incorrectly specified functions (e.g., non-smooth functions treated as being smooth) and adversarially perturbed functions.

Specifically, we experimentally evaluate the performance of agd and axgd under additive Gaussian noise. Fig. 2 illustrates the performance of agd and axgd when the gradients are corrupted by zero-mean additive Gaussian noise with covariance matrix $\epsilon_{\eta}I$, where $I$ is the identity matrix. When the region is unconstrained (top row in Fig. 2), both agd and axgd exhibit high sensitivity to noise. The gd method overall exhibits higher tolerance to noise (at the expense of slower convergence). In the case of the unit simplex region (bottom row in Fig. 2), all the algorithms appear more tolerant to noise than in the unconstrained case. Interestingly, on this example axgd exhibits higher tolerance to noise than gd and agd, both in terms of mean and in terms of variance. Explaining this phenomenon analytically is an interesting question that merits further investigation.

4 Conclusion

We have presented a novel accelerated method – axgd– that combines ideas from the Nesterov’s agd and Nemirovski’s mirror prox. axgd achieves optimal convergence rates for a range of convex optimization problems, such as the problems with the (i) smooth objectives, (ii) objectives with Hölder-continuous gradients, (iii) and non-smooth Lipschitz-continuous objectives. In the constrained-regime experiments from Section 3, the method demonstrates favorable performance compared to agd when subjected to zero-mean Gaussian noise.

There are several directions that merit further investigation. A more thorough analytical and experimental study of acceleration when the gradients are corrupted by noise is of particular interest, since the gradients can often come from noise-corrupted measurements. Further, our experiments from Fig. 2 suggest that there are cases that incur a trade-off between noise tolerance and acceleration. A systematic study of this trade-off is thus another important direction, since it would guide the choice of accelerated/non-accelerated algorithms in practice depending on the application. Finally, it is interesting to investigate whether restart schemes can improve the algorithms’ noise tolerance, since in the noiseless setting several restart schemes are known to improve the convergence of agd in practice.

Appendix A Properties of the Bregman Divergence

The following properties of Bregman divergence will be useful in our analysis.

Proposition A.1**.**

$D_{\psi}(\nabla\psi^{*}(\mathbf{z}),\mathbf{x})=D_{\psi^{*}}(\nabla\psi(\mathbf{x}),\mathbf{z})$, $\forall\mathbf{x},\mathbf{z}$.

Proof.

From the definition of $\psi^{*}$ and Fact 1.6,

[TABLE]

Similarly, as in the light of Fenchel-Moreau Theorem $\psi^{**}=\psi$,

[TABLE]

Using the definition of $D_{\psi}(\nabla\psi^{*}(\mathbf{z}),\mathbf{x})$ and Fact 1.6:

[TABLE]

Similarly, using the definition of $D_{\psi^{*}}(\nabla\psi(\mathbf{x}),\mathbf{z})$ combined with (A.1):

[TABLE]

Comparing (A.3) and (A.4), the proof follows. ∎

Proposition A.2**.**

If $\psi(\cdot)$ is $\sigma$-strongly convex, then $D_{\psi^{*}}(\mathbf{z},\mathbf{\hat{z}})\geq\frac{\sigma}{2}\|\nabla\psi^{*}(\mathbf{z})-\nabla\psi^{*}(\mathbf{\hat{z}})\|^{2}$.

Proof.

Using the definition of $D_{\psi^{*}}(\mathbf{z},\mathbf{\hat{z}})$ and (A.1), we can write $D_{\psi^{*}}(\mathbf{z},\mathbf{\hat{z}})$ as:

[TABLE]

Since $\psi(\cdot)$ is $\sigma$-strongly convex, it follows that:

[TABLE]

As, from Fact 1.6, $\nabla\psi^{*}(\mathbf{z})=\arg\max_{\mathbf{x}\in X}\{\left\langle\mathbf{x},\mathbf{z}\right\rangle-\psi(\mathbf{x})\}$, by the first-order optimality condition

[TABLE]

completing the proof. ∎

The Bregman divergence $D_{\psi^{*}}(\mathbf{x},\mathbf{y})$ captures the difference between $\psi^{*}(\mathbf{x})$ and its first order approximation at $\mathbf{y}.$ Notice that, for a differentiable $\psi^{*}$, we have:

[TABLE]

The Bregman divergence $D_{\psi^{*}}(\mathbf{x},\mathbf{y})$ is a convex function of $\mathbf{x}.$ Its Bregman divergence is itself.

Proposition A.3**.**

For all $\mathbf{x},\mathbf{y},\mathbf{z}\in X$

[TABLE]

Appendix B Omitted Proofs from Section 2

Proposition 2.1.

Let $\mathbf{z}^{(t)}=\nabla\psi(\mathbf{x}^{(t_{0})})-\int_{t_{0}}^{t}\nabla f(\mathbf{x}^{(\tau)})d\alpha^{(\tau)}$. Then:

[TABLE]

Proof.

From the definition of Bregman divergence:

[TABLE]

Using the definition of $\mathbf{z}^{(t)}$ and Fact 1.6, the proof follows. ∎

Remaining Proof of Theorem 2.5 (The Bound on $G_{1}$).

To bound $G_{1}$, we recall the definition of $L_{1}$:

[TABLE]

As $a_{1}=A_{1}$, $\mathbf{x}^{(1)}=\nabla\psi^{*}(\mathbf{\hat{z}}^{(0)})$, and $a_{1}\nabla f(\mathbf{\hat{x}}^{(0)})=\mathbf{z}^{(0)}-\mathbf{\hat{z}}^{(0)}$, using Proposition A.3, we have that:

[TABLE]

On the other hand, by smoothness of $f(\cdot)$ and the initial condition:

[TABLE]

Finally, by Proposition A.1 and the initial condition $\mathbf{z}^{(0)}=\nabla\psi(\mathbf{\hat{x}}^{(0)})$, we have that $D_{\psi^{*}}(\mathbf{z}^{(0)},\mathbf{z}^{(1)})=D_{\psi}(\nabla\psi^{*}(\mathbf{z}^{(1)}),\mathbf{\hat{x}}^{(0)})$. Combining with (B.1), (B.2), and $G_{1}=U_{1}-L_{1}=f(\mathbf{x}^{(1)})-L_{1}$:

[TABLE]

where we have used Proposition A.1, $\mathbf{x}^{(1)}=\nabla\psi^{*}(\mathbf{\hat{z}}^{(0)})$, and $\frac{{a_{1}}^{2}}{A_{1}}=A_{1}\leq\frac{\sigma}{L}$. ∎

Proposition 2.6.

[TABLE]

Proof.

From the first order optimality condition in Fact 1.6, $\forall\mathbf{x},\mathbf{y}\in X$:

[TABLE]

Letting $\mathbf{x}=\nabla\psi^{*}(\mathbf{\hat{z}}^{(i)})$, $\mathbf{y}=\nabla\psi^{*}(\mathbf{z}^{(i+1)})$, and summing (B.3) and (B.4):

[TABLE]

where (B.5) follows by the $\sigma$-strong convexity of $\psi(\cdot)$. Using the Cauchy-Schwartz inequality and dividing both sides by $\|\nabla\psi^{*}(\mathbf{\hat{z}}^{(i)})-\nabla\psi^{*}(\mathbf{z}^{(i+1)})\|$ gives $\|\mathbf{\hat{z}}^{(i)}-\mathbf{z}^{(i+1)}\|\geq\sigma\|\nabla\psi^{*}(\mathbf{\hat{z}}^{(i)})-\nabla\psi^{*}(\mathbf{z}^{(i+1)})\|$.

Since, by the step definition (2.2), $\mathbf{\hat{z}}^{(i)}-\mathbf{z}^{(i+1)}=a_{i+1}(\nabla f(\mathbf{x}^{(i+1)})-\nabla f(\mathbf{\hat{x}}^{(i)}))$, applying Cauchy-Schwartz Inequality to $a_{i+1}\left\langle\nabla f(\mathbf{x}^{(i+1)})-\nabla f(\mathbf{\hat{x}}^{(i)}),\nabla\psi^{*}(\mathbf{z}^{(i+1)})-\nabla\psi^{*}(\mathbf{\hat{z}}^{(i)})\right\rangle$ completes the proof. ∎