Nesterov's acceleration and Polyak's heavy ball method in continuous time: convergence rate analysis under geometric conditions and perturbations

TL;DR

This paper analyzes the convergence rates of a family of second order ODEs related to inertial gradient methods, considering geometric conditions and perturbations, to understand their behavior in optimization tasks.

Contribution

It provides the first joint analysis of geometrical properties and perturbations for these ODEs, offering new asymptotic bounds on convergence rates.

Findings

Derived new asymptotic bounds for convergence rates.

Analyzed effects of damping and perturbations on convergence.

Insights applicable to stochastic optimization algorithms.

Abstract

In this article a family of second order ODEs associated to inertial gradient descend is studied. These ODEs are widely used to build trajectories converging to a minimizer $x^*$ of a function $F$, possibly convex. This family includes the continuous version of the Nesterov inertial scheme and the continuous heavy ball method. Several damping parameters, not necessarily vanishing, and a perturbation term $g$ are thus considered. The damping parameter is linked to the inertia of the associated inertial scheme and the perturbation term $g$ is linked to the error that can be done on the gradient of the function $F$. This article presents new asymptotic bounds on $F(x(t))-F(x^*)$ where $x$ is a solution of the ODE, when $F$ is convex and satisfies local geometrical properties such as {\L}ojasiewicz properties and under integrability conditions on $g$. Even if geometrical properties and perturbations were already studied for most ODEs of these families, it is the first time they are jointly studied. All these results give an insight on the behavior of these inertial and perturbed algorithms if $F$ satisfies some {\L}ojasiewicz properties especially in the setting of stochastic algorithms.

Equations

$$\ddot{x}(t)+\beta (t)\dot{x}(t)+\nabla F(x(t))=g(t),$$

$${\int }_{t_0}^{+\infty }{t}^{\frac{1+\theta }{2}}\Vert g(t)\Vert dt<+\infty .$$

$${\int }_{t_0}^{+\infty }{t}^{\frac{\alpha }{3}}\Vert g(t)\Vert dt<+\infty$$

$$\begin{gathered} \forall x\in B(x^{\ast },\eta ),\text{leavevmode}F(x)-F^{\ast }⩽ \\ \frac{1}{\gamma }⟨\nabla F(x),x-x^{\ast }⟩. \end{gathered}$$

$$\forall x\in B(x^{\ast },ϵ),K_r\text{d}(x,X^{\ast }{)}^r⩽F(x)-F^{\ast }.$$

$$\forall x\in B(x^{\ast },\eta ),\text{leavevmode}F(x)-F(x^{\ast })\le M\Vert x-x^{\ast }{\Vert }^{\gamma }.$$

$$\forall x\in B(x^{\ast },\epsilon ),\text{leavevmode}\Vert \nabla F(x)\Vert ⩾c{\left(F(x)-F(x^{\ast })\right)}^{\theta }.$$

$$\forall x\in {\mathbb{R}}^n,F(x)-F^{\ast }⩾\frac{\mu }{2}d(x,X^{\ast }{)}^2,$$

$$F(x(t))-F^{\ast }=\mathcal{O}\left(t^{-\frac{2\gamma \alpha }{\gamma +2}}\right).$$

$$\ddot{x}(t)+\frac{\alpha }{t}\dot{x}(t)+\nabla F(x(t))=g(t).$$

$${\int }_{t_0}^{+\infty }{t}^{\frac{\gamma \alpha }{\gamma +2}}\Vert g(t)\Vert dt<+\infty .$$

$$F(x(t))-F^{\ast }=\mathcal{O}\left(t^{-\frac{2\gamma \alpha }{\gamma +2}}\right).$$

$$\ddot{x}(t)+\frac{\alpha }{t^{\theta }}\dot{x}(t)+\nabla F(x(t))=g(t).$$

$${\int }_{t_0}^{+\infty }{e}^{m\Gamma (t)}\Vert g(t)\Vert dt<+\infty .$$

$$F(x(t))-F^{\ast }=\mathcal{O}\left(e^{-m\Gamma (t)}\right),\Vert x(t)-x^{\ast }{\Vert }^2=\mathcal{O}\left(e^{-m\Gamma (t)}\right),\Vert \dot{x}(t){\Vert }^2=\mathcal{O}\left(e^{-m\Gamma (t)}\right).$$

$$F(x(t))-F^{\ast }=\mathcal{O}\left(\frac{1}{t^{\frac{2r{\gamma }_2}{{\gamma }_2-2}}}\right).$$

$${\int }_{t_0}^{+\infty }{t}^{\frac{r{\gamma }_2}{{\gamma }_2-2}}\Vert g(t)\Vert dt<+\infty .$$

$$F(x(t))-F^{\ast }=\mathcal{O}\left(\frac{1}{t^{\frac{2r{\gamma }_2}{{\gamma }_2-2}}}\right).$$

$${\int }_{t_0}^{+\infty }{t}^{\frac{r\gamma }{\gamma -2}}\Vert g(t)\Vert dt<+\infty .$$

$$\Vert \dot{x}(t)\Vert =\mathcal{O}\left(\frac{1}{t^{\frac{r\gamma }{\gamma -2}}}\right).$$

$$\underset{{\gamma }_2\in (2+\infty )}{\inf }\frac{(1+\theta ){\gamma }_2}{2({\gamma }_2-2)}=\frac{1+\theta }{2},$$

$$\mathcal{E}(t)=F(x(t))-F^{\ast }+\frac{1}{2}\Vert \dot{x}(t){\Vert }^2.$$

$$\mathcal{E}(t)=t^2(F(x(t))-F^{\ast })+\frac{1}{2}\Vert (\alpha -1)(x(t)-x^{\ast })+t\dot{x}(t){\Vert }_2^2.$$

$${\mathcal{E}}^{\prime }(t)⩽-\alpha t\Vert \dot{x}(t){\Vert }^2+(3-\alpha )t(F(x(t))-F^{\ast }).$$

$$F(x(t))-F^{\ast }⩽\frac{\mathcal{E}(t_0)}{t^2}.$$

$$\mathcal{H}(t)=t^p(F(x(t))-F^{\ast })+R(x(t)),$$

$${\mathcal{E}}^{\prime }(t)⩽-\gamma (t)\mathcal{E}(t)$$

$$\mathcal{E}(t)⩽\mathcal{E}(t_0)e^{-{\int }_{t_0}^t\gamma (s)ds}.$$

$$\mathcal{G}(t)=t^p\left(t^2(F(x(t))-F^{\ast })+\frac{1}{2}\Vert \lambda (x(t)-x^{\ast })+t\dot{x}(t){\Vert }_2^2+\frac{\xi }{2}\Vert x(t)-x^{\ast }{\Vert }^2\right)+{\int }_t^T{s}^{\delta }⟨\lambda (x(s)-x^{\ast })+s\dot{x}(s),g(s)⟩ds$$

$$\mathcal{G}(t)=F(x(t))-F^{\ast }+\frac{1}{2}\Vert \lambda (t)(x(t)-x^{\ast })+\dot{x}(t){\Vert }_2^2+\frac{\xi (t)}{2}\Vert x(t)-x^{\ast }{\Vert }^2+{\int }_t^T{s}^{\delta }⟨\lambda (x(s)-x^{\ast })+s\dot{x}(s),g(s)⟩ds.$$

Full text

Keywords

Lyapunov functions, rate of convergence, ODEs, optimization, Łojasiewicz property.

1. 1 Introduction

2. 2 Preliminaries: local geometry of convex functions

3. 3 Contributions

4. 3.1 Convergence rates for sharp geometries

5. 3.2 Convergence rates for flat geometries

6. 3.3 Strategies of proofs

7. 3.3.1 State of the art strategies

8. 3.3.2 Sketch of proof

9. 4 Proofs

10. 4.1 Proof of Theorem 3.2

11. 4.2 Proof of Theorem 3.3

12. 4.3 Proof of Theorem 3.4

13. 4.4 Proof of Theorem 3.5

14. 4.5 Proof of Corollary 3.9

15. A Appendix

16. A.1 Proof of Lemma 4.1: differentiating the energy function for Theorem 3.2

17. A.2 Proof of Lemma : differentiating the energy function of Theorem 3.3

18. A.3 Proof of Lemma : differentiating the energy function of Theorem 3.3 for the classical heavy ball system

19. A.4 Proof of Lemma 4.4: differentiating the energy function of Theorem 3.4

1 Introduction

Let $F:\mathbb{R}^{n}\rightarrow\mathbb{R}$ be a differentiable convex function admitting at least one minimizer. In this paper we study the asymptotic behavior of the trajectories of the perturbed second-order ordinary differential equation (ODE):

[TABLE]

where $t_{0}>0$, $\beta(t)=\dfrac{\alpha}{t^{\theta}}$, with $\alpha>0$ and $\theta\in[0,1]$, is a viscous damping coefficient and $g:[t_{0},+\infty[\rightarrow\mathbb{R}$ an integrable source term that can be interpreted as a small external perturbation exerted on the system. Throughout the paper, we assume that, for any initial conditions ($x_{0},v_{0})\in\mathbb{R}^{n}\times\mathbb{R}^{n}$, the Cauchy problem associated with the differential equation (1.1), has a unique global solution satisfying $(x(t_{0}),\dot{x}(t_{0}))=(x_{0},v_{0})$. This is guaranteed for instance when the gradient function $\nabla F$ is Lipschitz on bounded subsets of $\mathbb{R}^{n}$ [19, 20].

During the last five years many articles study these ODEs, the convergence of the trajectory $x(t)$ or the decay rate of $F(x(t))$ to its minimum value $F^{*}$, see for example [8, 25, 23] and reference therein. In [25] Su et al. proved the Nesterov acceleration scheme can be seen as a discretization scheme of the ODE (1.1) with $\theta=1$ and $g(t)=0$. Moreover the convergence properties of the solution $x(t)$ of (1.1) are directly linked with the ones of the sequence defined by the Nesterov scheme and the Lyapunov analysis used in both cases to prove the convergence are very similar. As another example, the choice $\theta=0$ corresponds to the heavy ball damping. It turns out that this family of ODEs is related to inertial optimization algorithms, with various inertia, depending on the choice of the damping function $\frac{\alpha}{t^{\theta}}$ and including perturbation or error terms defined by $g$. Many results concerning inertial algorithms have been transposed to the continuous setting such as the convergence of FISTA iterates in [17] by Chambolle et al. which has been transposed by Attouch et al. [6] to the weak convergence of the trajectory of the solution of (1.1) with $\theta=1$ and $\alpha>3$. Conversely May in [23] and Attouch et al. in [6] proved that for $\theta=1$ and $\alpha>3$, if $F$ is convex, the solution $x$ of (1.1) satisfies $F(x(t))-F^{*}=o\left(\frac{1}{t^{2}}\right)$ and this result has been extended to the sequence generated by FISTA by Attouch et al. in [8]. Consequently, studying (1.1) is also a first step to have a better understanding of general and perturbed inertial schemes to minimize convex functions.

In [15] Cabot et al. consider a general damping term and a vanishing perturbation term $g=0$. Their study gives decay rates on $F(x(t))-F^{*}$ when $\theta\in[0,1)$. If only a convexity assumption is made on $F$, Su et al [25] proved that if $\theta=1$ and $\alpha\geqslant 3$, we can get $F(x(t))-F^{*}=O\left(\frac{1}{t^{2}}\right)$. In [6, 3] authors complete these first results for $\theta=1$, when $\alpha>3$, proving the weak convergence of trajectory $x$ and showing that $F(x(t))-F^{*}=o\left(\frac{1}{t^{2}}\right)$. In [9, 7] authors give some optimal bound on $F(x(t))-F^{*}$ in the subcritical case $\alpha<3$. More general damping functions $\beta(t)$ have been studied by Cabot et al., Jendoubi et al., Attouch et al. see [16, 20, 3] for complete results. In particular, if $F$ is convex and $\theta\in[0,1)$, we can get: $F(x(t))-F^{*}=O\left(\frac{1}{t^{1+\theta}}\right)$.

Several works extend these previous results with a non vanishing perturbation term $g$ proposing some integrability conditions on $g$, see for example Balti et al. [11] for $\theta\in[0,1)$ and Attouch et al. [6] for $\theta=1$ and $\alpha\geqslant 3$. In these two settings, the condition on $g$ ensuring the optimal decay rate $F(x(t))-F^{*}=O\left(\frac{1}{t^{1+\theta}}\right)$ is the following:

[TABLE]

For $\theta=1$ and $\alpha<3$, Attouch et al. and Aujol et al. [7, 9] proved that this condition can be weakened to

[TABLE]

to ensure that $F(x(t))-F^{*}=O\left(\frac{1}{t^{\frac{2\alpha}{3}}}\right)$. In [3] Attouch et al. for $\theta\in(0,1]$ and in [10] Aujol et al. for $\theta=1$ proved that these decay rates can be improved if more geometrical properties are known on $F$ when the perturbation term $g$ vanishes. These geometrical properties describe the growth of $F$ around the set of minimizers and are linked with Łojasiewicz properties when $F$ is convex.

The goal of this work is to generalize all the previous works providing accurate rates on $F(x(t))-F^{*}$ for any $\alpha>0$, for any $\theta\in[0,1]$, depending on the geometrical properties of $F$ such as Łojasiewicz properties and integrability conditions on $g$. To our best knowledge, this is the first work combining geometrical properties on $F$ and integrability on $g$ to provide decays on $F(x(t))-F^{*}$. More precisely, we will always consider that $F$ is convex, has a unique minimizer and we always assume integrability conditions on $g$ that ensure the convergence of $F(x(t))-F^{*}$ to [math]. Consequently, the convergence of the trajectory $(x(t))_{t\geqslant t_{0}}$ to the unique minimizer is always ensured. That is why, in all theorems, the geometrical assumptions are only made on a neighborhood of the minimizer and are not necessarily global.

The paper is organized as follows. In Section 2, we introduce the geometrical hypotheses we consider on the function $F$, and their relation with Łojasiewicz property. We then present the contributions of the paper in Section 3: depending on the geometry of the function $F$ and the value of the damping parameters $\alpha$ and $\theta$, we show that combining a flatness condition and a sharpness condition such as the Łojasiewicz property provides new and better convergence rates for the values $F(x(t))-F^{*}$. The proofs of the theorems are given in Section 4. Some technical proofs are postponed to Appendix A.

2 Preliminaries: local geometry of convex functions

In this section we recall some definitions and results concerning the local geometry of convex functions around their set of minimizers, see [10] for more details.

Throughout the paper, we assume that the ODE (1.1) is defined in $\mathbb{R}^{n}$ equipped with the euclidean scalar product $\langle\cdot,\cdot\rangle$ and the associated norm $\|\cdot\|$. As usual $B(x^{*},r)$ denotes the open euclidean ball with center $x^{*}\in\mathbb{R}^{n}$ and radius $r>0$. We now introduce on the one hand a flatness assumption that ensures that the function is not too sharp in the neighborhood of its minimizers, and on the other hand a sharpness assumption ensuring that the magnitude of the gradient is not too low in the neighborhood of the minimizers.

Definition 2.1**.**

Let $F:\mathbb{R}^{n}\rightarrow\mathbb{R}$ be a convex differentiable function with $X^{*}=\operatorname*{argmin}F\neq\emptyset$, and $F^{*}=\inf f$.

Let $\gamma\geqslant 1$. The function $F$ satisfies the condition $\mathbf{H_{1}}(\gamma)$ if, for any minimizer $x^{*}\in X^{*}$, there exists $\eta>0$ such that:

[TABLE] 2. 2.

Let $r\geqslant 1$. The function $F$ satisfies the growth condition $\mathbf{H_{2}}(r)$ if for any minimizer $x^{*}\in X^{*}$, there exist $K_{r}>0$ and $\epsilon>0$ such that:

[TABLE]

The assumption $\mathbf{H}_{1}(\gamma)$ has been already used in [15, 25, 9, 10, 1]. Note that any convex differentiable function satisfies $\mathbf{H}_{1}(1)$ and that any differentiable function such that $(F-F^{*})^{\frac{1}{\gamma}}$ is convex for some $\gamma\geqslant 1$, satisfies $\mathbf{H_{1}}(\gamma)$. More precisely, the hypothesis $\mathbf{H}_{1}(\gamma)$ can be seen as a flatness condition on the geometry of a convex function around its sets of minimizers [10, Lemma 2.4]: any convex differentiable function $F$ satisfying $\mathbf{H}_{1}(\gamma)$ for some $\gamma\geqslant 1$, also satisfies: for any minimizer $x^{*}\in X^{*}$, there exist $M>0$ and $\eta>0$ such that:

[TABLE]

The hypothesis $\mathbf{H}_{2}(r)$ with $r\geq 1$, is a growth condition on the function $F$ around its set of minimizers (critical points in the non-convex case) ensuring that $F$ is sufficiently sharp (at least as sharp as $x\mapsto\|x-x^{*}\|^{r}$) in the neighborhood of $X^{*}$. It is also called r-conditioning [18] or Hölderian error bounds [13]. In the convex setting, this growth condition is equivalent to the Łojasiewicz inequality [21, 22], a key tool in the mathematical analysis of continuous and discrete dynamical systems, with exponent $\theta=1-\frac{1}{r}\in(0,1]$:

Definition 2.2**.**

A differentiable function $F:\mathbb{R}^{n}\to\mathbb{R}$ is said to have the Łojasiewicz property with exponent $\theta\in[0,1)$ if, for any critical point $x^{*}$, there exist $c>0$ and $\varepsilon>0$ such that:

[TABLE]

where: $0^{0}=0$ when $\theta=0$ by convention.

Typical examples of functions having the Łojasiewicz property are real-analytic functions and $C^{1}$ subanalytic functions, or semi-algebraic functions [21, 22]. Strongly convex functions satisfy a global Łojasiewicz property with exponent $\theta=\frac{1}{2}$ [2], or equivalently a global version of the growth condition, namely:

[TABLE]

where $\mu>0$ denotes the parameter of strong convexity. Likewise, convex functions having a strong minimizer in the sense of [4, Section 3.3], also satisfy a global version of $\mathbf{H}_{2}(2)$. By extension, uniformly convex functions of order $p\geqslant 2$ satisfy the global version of the hypothesis $\mathbf{H}_{2}(p)$ [18].

Finally, observe that any convex differentiable function $F$ satisfying both hypothesis $\mathbf{H}_{1}(\gamma)$ and $\mathbf{H}_{2}(r)$, has to be at least as flat as $\|x-x^{*}\|^{\gamma}$ and as sharp as $\|x-x^{*}\|^{r}$ in the neighborhood of its minimizers. More precisely, combining (2.1) and $\mathbf{H}_{2}(r)$, we have:

Lemma 2.1** ([10, Lemma 2.5]).**

If a convex differentiable function $F$ satisfies both $\mathbf{H}_{1}(\gamma)$ and $\mathbf{H}_{2}(r)$, with $\gamma,r\geqslant 1$, then necessarily: $r\geqslant\gamma$.

3 Contributions

In this section, we state convergence rates for the values $F(x(t))-F^{*}$ along the trajectory $x(t)$ solution of (1.1), depending on geometrical properties $\mathbf{H}_{1}$ and $\mathbf{H}_{2}$ of the function $F$ and on integrability conditions on the perturbation term $g$. Geometry and perturbations have been studied separately in several papers for this family of ODEs, and the specificity of this work is to study both aspects jointly.

Consider first the case when $\theta=1$. If $F$ only satisfies $\mathbf{H}_{1}(\gamma)$ for some $\gamma\geqslant 1$ then for low friction parameter $\alpha$, we have the following result:

Theorem 3.1** ([9, Theorem 2]).**

Let $\alpha>0$ and $t_{0}>0$. Let $x(\cdot)$ be any solution of the ODE (1.1) with $\theta=1$ and $(x(t_{0}),\dot{x}(t_{0}))=(x_{0},v_{0})$. Assume that: $\int_{t_{0}}^{+\infty}t^{\frac{\gamma\alpha}{\gamma+2}}\|g(t)\|dt<+\infty.$ If $F$ satisfies $\mathbf{H}_{1}(\gamma)$ for some $\gamma\geqslant 1$ and if $\alpha\leqslant 1+\frac{2}{\gamma}$ then:

[TABLE]

This result has been first stated and proved in the unpublished report [9, Theorem 2] by Aujol and Dossal in 2017 for convex differentiable functions satisfying $(F-F^{*})^{\frac{1}{\gamma}}$ convex.

For large friction parameters $\alpha$, the sole assumption $\mathbf{H}_{1}(\gamma)$ on $F$ is not sufficient anymore to obtain a decay faster than $\mathop{}\mathopen{}{\large\mathcal{O}}\mathopen{}\left(\frac{1}{t^{2}}\right)$ which is the uniform rate that can be achieved for $\alpha\geqslant 3$ [25]. The contribution of this paper is to show that a flatness condition $\mathbf{H}_{1}$ associated to a sharpness condition such as the Łojasiewicz property provides new and better convergence rates for the values $F(x(t))-F^{*}$. We can thus compare these results with classical bounds that can be achieved with geometrical assumptions on $F$, such as convexity or Łojasiewicz properties without any perturbation term or with results dealing with a non vanishing perturbation term $g$ but with simple assumptions on $F$.

3.1 Convergence rates for sharp geometries

In this section, we state convergence rates on the values $F(x(t))-F^{*}$ along the trajectory $x(t)$, that can be achieved for functions satisfying geometrical hypothesis such as $\mathbf{H}_{1}(\gamma)$ and/or $\mathbf{H}_{2}(2)$. The cases $\theta=1$ and $\theta\in[0,1)$ are treated separately.

Let us first consider the case when $\theta=1$ i.e. the ODE:

[TABLE]

Theorem 3.2**.**

Let $\alpha>0$ and $t_{0}>0$. Let $x(\cdot)$ be any solution of the ODE (3.2) with $\theta=1$ and $(x(t_{0}),\dot{x}(t_{0}))=(x_{0},v_{0})$. Assume that:

[TABLE]

If $F$ satisfies $\mathbf{H}_{1}(\gamma)$ and $\mathbf{H}_{2}(2)$, for some $\gamma\leqslant 2$, if $F$ has a unique minimizer and if $\alpha>1+\frac{2}{\gamma}$, then

[TABLE]

Note that Theorem 3.2 whose proof is detailed in Section 4.1, only applies for $\gamma\leqslant 2$ since according to Lemma 2.1, there exists no function satisfying both $\mathbf{H}_{1}(\gamma)$ and $\mathbf{H}_{2}(2)$ for $\gamma>2$. Moreover the integrability condition given in Theorem 3.2 generalizes the integrability condition given in [9] to any $\alpha>0$ under the growth condition $\mathbf{H}_{2}(2)$ and coincides in the limit case $\alpha=3$, which was expected. Theorems 3.2 can be seen as an extension of former results with a non vanishing perturbation term $g$, see [10, Theorem 4.2].

Let $\alpha>0$ and $\theta\in[0,1)$. We now consider the heavy ball system with a general friction term:

[TABLE]

Theorem 3.3**.**

Let $\gamma\in[1,2]$, $m\in\left(0,\frac{2\gamma}{\gamma+2}\right)$ and $t_{0}>0$. Note: $\Gamma(t)=\int_{t_{0}}^{t}\frac{\alpha}{s^{\theta}}ds$. Let $x(\cdot)$ be any solution of the ODE (3.5) with $\theta\in[0,1)$ and $(x(t_{0}),\dot{x}(t_{0}))=(x_{0},v_{0})$. Assume that:

[TABLE]

If $F$ satisfies $\mathbf{H}_{1}(\gamma)$ and $\mathbf{H}_{2}(2)$, and admits a unique minimizer $x^{*}$ then:

[TABLE]

Theorem 3.3 when $\theta\in(0,1)$ can be seen as an extension of [3, Theorem 6.1] to functions with some geometrical properties as $\mathbf{H}_{1}$ and $\mathbf{H}_{2}$. Its proof is detailed in Section 4 and is an extension of the proof of [3, Theorem 3.12] to a non-vanishing perturbation term $g\neq 0$. Note that [3] deals only with vanishing damping, that is $\theta>0$, while Theorem 3.3 deals also with the case $\theta=0$.

Observe also that Theorem 3.3 also applies in the case $\theta=1$ and provides convergence rates in $\mathop{}\mathopen{}{\large\mathcal{O}}\mathopen{}\left(t^{-m\alpha}\right)$ for any $m\in\left(0,\frac{2\gamma}{\gamma+2}\right)$, which is slower but infinitely close to the convergence rate $\mathop{}\mathopen{}{\large\mathcal{O}}\mathopen{}\left(t^{-\frac{2\gamma\alpha}{\gamma+2}}\right)$ provided by Theorem 3.2.

Finally, observe that the integrability conditions given in these three theorems are always stronger than the condition $\int_{t_{0}}^{+\infty}t^{p}\|g(t)\|dt<+\infty$, with $p=\min(1,\frac{\alpha}{3})$, given in [7, Theorem 5.1] and ensuring the convergence of the values $F(x(t))-F^{*}$ to [math]. Consequently the trajectory $x(t)$ actually converges to the unique minimizer of $F$ so that the geometrical assumptions $\mathbf{H}_{1}$ and $\mathbf{H}_{2}$ can be used locally that is in the neighborhood of the unique minimizer of $F$.

3.2 Convergence rates for flat geometries

In this section, we state new convergence rates on the values $F(x(t))-F^{*}$ along the trajectory $x(t)$, that can be achieved for functions satisfying geometrical hypothesis such as $\mathbf{H}_{1}(\gamma)$ and $\mathbf{H}_{2}(\gamma)$ for any $\gamma>2$. The cases $\theta=1$ and $\theta\in[0,1)$ are treated jointly.

Let us first consider the unperturbed case ($g=0$). We announce new results on the convergence of the function values along the trajectory $x(t)$ with additional geometrical assumptions, but without perturbations:

Theorem 3.4**.**

Let $\gamma_{1}>2$, $\gamma_{2}\geqslant\gamma_{1}$. Note $r=\frac{1+\theta}{2}$. Suppose $x$ is a solution to the ODE (1.1) with $g=0$. If $F$ is coercive and satisfies $\mathbf{H_{1}}(\gamma_{1})$ and $\mathbf{H_{2}}(\gamma_{2})$, and

if $\theta=1$ and $\alpha\geqslant\frac{\gamma_{1}+2}{\gamma_{1}-2}$ or 2. 2.

if $\theta<1$,

then:

[TABLE]

Note that Theorem 3.4 is a generalization of Theorem 4.3 in [10] available for $\theta=1$ to any $\theta\in(0,1]$. We now prove that the convergence rates provided by Theorem 3.4 remain valid in the perturbed case ($g\neq 0$):

Theorem 3.5**.**

Let $\alpha>0$, $\theta\in[0,1]$ and $t_{0}>0$. Let $x$ be the solution of the ODE (1.1) for given initial conditions $(x(t_{0}),\dot{x}(t_{0}))=(x_{0},v_{0})$. Let $\gamma_{1}>2$, $\gamma_{2}\geqslant\gamma_{1}$ and $r=\frac{1+\theta}{2}$. Assume that:

[TABLE]

If $F$ satisfies $\mathbf{H_{1}}(\gamma_{1})$ and $\mathbf{H_{2}}(\gamma_{2})$ and admits a unique minimizer then:

if $\theta=1$ and $\alpha\geqslant\frac{\gamma_{1}+2}{\gamma_{1}-2}$ or 2. 2.

if $\theta<1$,

then we have

[TABLE]

As in [10] with a non vanishing perturbation term $g$, if $\gamma_{1}=\gamma_{2}$, we have furthermore the convergence of the trajectory:

Corollary 3.1**.**

Let $\alpha>0$, $\theta\in[0,1]$ and $t_{0}>0$. Let $x$ be the solution of the ODE (1.1) for given initial conditions $(x(t_{0}),\dot{x}(t_{0}))=(x_{0},v_{0})$. Let $\gamma>2$. Note $r=\frac{1+\theta}{2}$.Assume that:

[TABLE]

If $F$ satisfies $\mathbf{H}_{1}(\gamma)$ and $\mathbf{H}_{2}(\gamma)$ and admits a unique minimizer then:

if $\theta=1$ and $\alpha\geqslant\frac{\gamma+2}{\gamma-2}$ or 2. 2.

if $\theta<1$,

then we have:

[TABLE]

Note that in the case of the classical heavy ball ($\theta=0$), Theorem 3.5 can be seen as an extension of [12, Corollary 5.1] using a different approach: indeed in [12], the authors proved a similar convergence rate under Łojasiewicz properties, but without any convexity assumption on $F$.

Observe also that to deal with a non-vanishing perturbation term, the uniqueness of the minimizer seems to be crucial despite the fact we can avoid this assumption when $g=0$, see [10, Theorem 4.5 and Corollary 4.6].

Finally observe that in the integrability condition given in Theorem 3.5, the exponent is $\frac{r\gamma_{2}}{\gamma_{2}-2}=\frac{(1+\theta)\gamma_{2}}{2(\gamma_{2}-2)}$. Since $\frac{r\gamma_{2}}{\gamma_{2}-2}\geqslant r$, the integrability condition (1.2) from [23] is automatically satisfied and ensures that the trajectory $x(t)$ converges to the unique minimizer of $F$. The geometrical assumptions $\mathbf{H}_{1}$ and $\mathbf{H}_{2}$ thus can be used locally. Moreover:

[TABLE]

so that we get the same exponent as in the integrability condition in (1.2) which is also what we expected.

3.3 Strategies of proofs

The guideline of the proofs of our results is the same: Lyapunov functions (or energies). In this section we present the state of the art strategies using Lyapunov functions and a sketch of the strategies used in this paper.

Note that in the four Theorems 3.1, 3.2, 3.3 and 3.5 dealing with a non vanishing perturbation term, we modified Lyapunov functions used when $g=0$, and proposed integrability conditions on $g$ that ensure the same decay that the one that can be achieved with $g=0$. That is the reason why the Lyapunov functions we use are closed to those that can be found in the literature. Note also that the Grönwall-Bellman Lemma is a key lemma in each proof, but the exact way to deal with a non vanishing perturbation term $g$ is different in each theorem.

3.3.1 State of the art strategies

To prove each decay of $F(x(t))-F^{*}$, the main idea is to define a function of $t$ which will be denoted by $\mathcal{E}$, $\mathcal{H}$ or $\mathcal{G}$, which involves the term $F(x(t))-F^{*}$ and which is bounded. The choice of the Lyapunov function depends on the ODE ($\alpha$ and $\theta$) and on assumptions on function $F$ (the flatness hypothesis $\mathbf{H}_{1}$). A simple Lyapunov function to study solutions of (1.1) when $F$ is convex with $g=0$ is:

[TABLE]

Indeed, $\mathcal{E}$ is a sum of positive terms and $\mathcal{E}^{\prime}(t)=-\beta(t)\|\dot{x}(t)\|^{2}\leqslant 0$. This simple Lyapunov function ensures that $\mathcal{E}$ is non increasing which implies that $F(x(t))$ is bounded. Many Lyapunov have been proposed to study (3.2) when $F$ is convex, see for example [25, 5, 7, 9] or for more general friction term, see [15, 11]. A simple example to study the specific case of Nesterov damping (i.e. $\theta=1$) when $F$ is convex and $g=0$, is:

[TABLE]

Indeed, a simple calculation shows that:

[TABLE]

From this Lyapunov function we deduce that if $\alpha\geqslant 3$, $\mathcal{E}$ is non increasing and thus that

[TABLE]

In [10], we propose to extend this Lyapunov approach to deal with geometrical properties of $F$ for the Nesterov damping i.e (3.2) with $g=0$ using Lyapunov functions $\mathcal{H}$ :

[TABLE]

where $R$ is non necessarily positive and $p$ depends on properties of the damping parameter $\alpha$ and on the geometric properties of $F$ to get $F(x(t))-F^{*}=O\left(\frac{1}{t^{p}}\right)$. To get such a bound we first bound the energy $\mathcal{H}$ and thus use the Łojasiewicz properties of $F$ ($\mathbf{H}_{2}$) to deduce the bound on $F(x(t))-F^{*}$.

For other choices of function $\beta(t)$, especially when $\theta<1$ in (1.1), the decay may be faster than polynomial if $F$ satisfies some Łojasiewicz properties, see [24, 3, 13]. A way to prove these faster decay is to build Lyapunov energies satisfying some differential inequalities for suitable function $\gamma$ :

[TABLE]

which implies that

[TABLE]

Moreover, a simple way to deal with the perturbation term $g$ in (1.1) is to add an integral term in the Lyapunov energy $\mathcal{E}$ or $\mathcal{H}$ depending on $x$ and $g$, see for example [11, 5, 9] and references therein and to use a Grönwall-Bellman lemma to conclude.

3.3.2 Sketch of proof

For each theorem, we define a Lyapunov Energy $\mathcal{G}$ (or $\mathcal{H}$) defined by parameters that are set depending on the hypotheses of each theorem. This function $\mathcal{G}$ also depends on the perturbation term $g$. The first step of the proof consists in proving that $\mathcal{G}$ is bounded. The second step use a Grönwall-Bellman lemma and hypothesis $\mathbf{H}_{2}$ to conclude. More precisely, Theorems 3.2 and 3.3 deal with sharp functions, that is functions satisfying $\mathbf{H}_{2}(2)$, or Łojasiewicz properties with parameter equal to $\frac{1}{2}$. The first one is dedicated to Nesterov damping ($\theta=1$). The polynomial rate was known in the case where $g=0$, see [10], and we propose to modify the Lyapunov function that is used by adding an integral term :

[TABLE]

and prove that $\mathcal{G}$ is bounded using differential inequalities. The Grönwall-Bellman lemma and integrability hypotheses on $g$ are used to conclude.

Theorem 3.3 deals with the case $\theta<1$. In this case it is known [24, 3, 13] that the decay of $F(x(t))-F^{*}$ is faster than polynomial. We propose to use a Lyapunov function similar to the previous one. This time $\lambda$ and $\xi$ may be functions of $t$ :

[TABLE]

For suitable choices of $\lambda,\xi$ and $\delta$ we can get some differential inequalities like (3.15) satisfied by $\mathcal{G}$ and conclude using Grönwall-Bellman lemma.

Theorems 3.4 and 3.5 deal with flat functions that is functions $F$ satisfying $\mathbf{H}_{2}(\gamma_{2})$ with $\gamma_{2}>2$, i.e Łojasiewicz properties with an exponent greater than $\frac{1}{2}$. Theorem 3.4 focuses on the case $g=0$ and Theorem 3.5 is the general perturbed case. Both theorems provide results for $\theta\in[0,1]$ including Heavy Ball ($\theta=0$) and Nesterov ($\theta=1$). In both theorems, the bound given on $F(x(t))-F^{*}$ is polynomial. For Theorem 3.4, inspired by [10] we define an energy function

[TABLE]

for a suitable choice of parameters $\lambda$ and $p$, and function $\xi$. And we prove that it exists $t_{1}\geqslant t_{0}$ such that for any $t\geqslant t_{1}$, $\mathcal{H}^{\prime}(t)\leqslant 0$. The function $\xi$ may be negative but using Łojasiewicz properties of $F$, we can deduce bounds on $F(x(t))-F^{*}$. In Theorem 3.5, we consider:

[TABLE]

and prove that $\mathcal{G}$ is bounded. Combining approaches developed in Theorem 3.4, Grönwall-Bellman lemma and integrability hypotheses on $g$, we are able to conclude.

4 Proofs

In this section, we detail the proofs of Theorem 3.2, Theorem 3.3, Theorem 3.4, Theorem 3.5 and Corollary 3.9.

4.1 Proof of Theorem 3.2

In this section, we prove Theorem 3.2. For a complete proof of Theorem 3.1, we refer the reader to [9].

Let $x^{*}$ be a minimizer of $F$ and $\lambda$, $\xi$ and $T$ three real numbers. The proof of Theorem 3.2 relies on the following energy:

[TABLE]

where the energy $\mathcal{E}$ is defined by:

[TABLE]

Using the following notations:

[TABLE]

we then have:

[TABLE]

Note that the functions $\mathcal{E}(t)$ and $\mathcal{H}(t)=t^{p}\mathcal{E}(t)$ denote the same Lyapunov functions as those used in the proof of Theorem 4.1. in [10] in the non perturbed case ($g=0$). Our proofs are based on the following lemma:

Lemma 4.1**.**

Let $\gamma\geqslant 1$. If $F$ satisfies the hypothesis $\mathbf{H}_{1}(\gamma)$ and if $\xi=\lambda(\lambda+1-\alpha)$, then

[TABLE]

This lemma whose proof is detailed in Appendix A.1, is the generalization of [10, Lemma 5.1] to the perturbed case: the integral term in (4.1) was chosen to cancel the terms in $\mathcal{G}^{\prime}(t)$ coming from the perturbation $g(t)$: following the exact same calculation steps as in the proof of [10, Lemma 5.1], we observe that all the terms coming from the perturbation $g(t)$ cancel each other out, so that we obtain the same formula as in the non perturbed version.

Choosing now $p=\frac{2\gamma\alpha}{\gamma+2}-2$ and $\lambda=\frac{2\alpha}{\gamma+2}$ and thus:

[TABLE]

we get:

[TABLE]

where $K_{1}=\xi(p-2\lambda)=\frac{2\xi}{\gamma+2}\left((\gamma-2)\alpha-(\gamma+2)\right)$. Since $\alpha>1+\frac{2}{\gamma}$ and $\gamma\leqslant 2$, we necessarily have: $\xi<0$, and thus $K_{1}>0$. Consequently, the energy $\mathcal{E}(t)$ is not a sum of non-negative terms and we cannot conclude that the energy function $\mathcal{G}$ is decreasing. To get the expected estimate on the energy, we need an additional growth condition $\mathbf{H}_{2}(2)$ to bound the term $\|x(t)-x^{*}\|^{2}$ as done in [10].

Using the uniqueness of the minimizer and the fact that $F$ satisfies $\mathbf{H}_{2}(2)$, there exists $K>0$ such that:

[TABLE]

hence:

[TABLE]

Since $\xi<0$ with our choice of parameters, we get:

[TABLE]

so that there exists $t_{1}\geqslant t_{0}$ such that for all $t\geqslant t_{1}$, we have:

[TABLE]

Now from (4.6), (4.8) and (4.10), we have:

[TABLE]

Observe now that: $\mathcal{H}^{\prime}(t)=\mathcal{G}^{\prime}(t)+\langle\lambda(x(t)-x^{*})+t\dot{x}(t),t^{p+1}g(t)\rangle$ so that we get the following differential inequality on the energy $\mathcal{H}$:

[TABLE]

Using again the fact that $c(t)=\mathop{}\mathopen{}{\scriptstyle\mathcal{O}}\mathopen{}\left(a(t)\right)$ (see (4.8)), there exists $t_{2}\geqslant t_{1}$ such that, for all $t\geqslant t_{2}$, $a(t)+\xi c(t)\geqslant\frac{1}{2}a(t)$, which implies that:

[TABLE]

Hence:

[TABLE]

Dividing both sides of the inequality by $2\mathcal{H}(t)^{\frac{1}{2}}$ and integrating between $t_{2}$ and $t$, we get:

[TABLE]

Since $\int_{t_{2}}^{+\infty}s^{\frac{\gamma\alpha}{\gamma+2}}\|g(s)\|ds<+\infty$:

[TABLE]

where $\beta=\frac{\sqrt[]{2}}{2}\displaystyle\int_{t_{2}}^{+\infty}s^{\frac{\gamma\alpha}{\gamma+2}}\|g(s)\|ds$. Applying the Grönwall Lemma, we finally get:

[TABLE]

In other words, we found a constant $A>0$ such that for all $t\geqslant t_{2}$, $\mathcal{H}(t)\leqslant A$. According to (4.10), we conclude that $\frac{1}{2}t^{p+2}(F(x(t))-F^{*})=\frac{1}{2}t^{p+1}a(t)$ is bounded which ends the proof of Theorem 3.2.

4.2 Proof of Theorem 3.3

Let $\lambda$, $\xi$ and $T$ three real numbers. Let $x^{*}$ be a minimizer of $F$ and $x(\cdot)$ any trajectory solution of:

[TABLE]

where: $\beta(t)=\frac{\alpha}{t^{\theta}}$ with $\alpha>0$ and $\theta\in[0,1)$. The proof of Theorem 3.3 relies on the following energy:

[TABLE]

where:

[TABLE]

Case $\theta=0$.

Remember that in that case, the friction coefficient is constant: $\forall t,\leavevmode\nobreak\ \beta(t)=\alpha$. Using the following notations:

[TABLE]

the energy $\mathcal{E}(t)$ can be rewritten as:

[TABLE]

The proof of Theorem 3.3 when $\theta=0$ relies on the following differential inequality whose proof in detailed in appendix:

Lemma 4.2**.**

Let $\gamma\geqslant 1$, $\theta=0$ and $\lambda\in\mathbb{R}$. If $F$ satisfies the hypothesis $\mathbf{H}_{1}(\gamma)$ and $\xi=\lambda(\lambda-\alpha)$, then:

[TABLE]

The scheme of the rest of the proof is quite standard: we first need to control the terms in $b(t)$ and $c(t)$ in Lemma 4.2 to deduce some differential inequality on the energy $\mathcal{E}$.

Let us choose $\lambda<\alpha$. In that case, the energy $\mathcal{E}$ is not a sum of non-negative terms anymore:

[TABLE]

since: $\xi=\lambda(\lambda-\alpha)<0$. Using the growth condition $\mathbf{H}_{2}(2)$ combined by the uniqueness of the minimizer of $F$, to bound $\|x(t)-x^{*}\|$, we get the following inequality: there exists $t_{1}\geqslant t_{0}$ such that:

[TABLE]

From now on, we choose: $\lambda=\frac{\gamma K_{2}}{2\alpha}$. Observe that the constant $K_{2}$ appearing in the growth condition $\mathbf{H}_{2}(2)$ can be chosen as small as needed to get: $\lambda\leq\frac{2}{\gamma+2}\alpha<\alpha$. With that choice, we then have:

[TABLE]

and $2(\lambda-\alpha)\leqslant-\lambda\gamma$. It follows from (4.20) that, for all $t\geq t_{1}$:

[TABLE]

and, noticing that $a(t)+b(t)\geqslant\mathcal{E}(t)$, we finally get for all $t\geq t_{1}$:

[TABLE]

or equivalently:

[TABLE]

The rest of the proof is quite standard: before integrating the differential inequality (4.22) between $t_{1}$ and $t$, we first need to control the term $\|\dot{x}(t)+\lambda(x(t)-x^{*})\|$. To that end, observe that, according to (4.23), the energy $\mathcal{G}$ is non-increasing, hence: $\forall t\geqslant t_{1},\leavevmode\nobreak\ \mathcal{G}(t)\leq\mathcal{G}(t_{1}),$ i.e.:

[TABLE]

With our choice of parameters, the energy $\mathcal{E}$ is not a sum of non-negative terms, so that the term $\|\dot{x}(s)+\lambda(x(s)-x^{*})\|$ can not be directly controlled by $\mathcal{E}(t)$. But according to the growth condition $\mathbf{H}_{2}(2)$ and to the uniqueness of the minimizer, we have for all $t\geqslant t_{1}$: $c(t)\leqslant\frac{1}{2K_{2}}a(t)$. Using (4.21) and $\gamma\leqslant 2$, we deduce:

[TABLE]

hence: $\forall t\geqslant t_{1},\leavevmode\nobreak\ \mathcal{E}(t)\geqslant\frac{1}{2}a(t)+b(t)\geqslant b(t)$. It follows:

[TABLE]

Applying the Grönwall-Bellman Lemma [14, Lemma A.5], we obtain:

[TABLE]

where: $c=\sqrt{2\mathcal{E}(t_{1})}$. Since $\int_{t_{1}}^{+\infty}\|g(s)\|ds<+\infty$ by assumption, we can conclude that:

[TABLE]

Coming back to (4.22), we obtain the following differential inequality:

[TABLE]

Integrating between $t_{1}$ and $t$ , we finally obtain:

[TABLE]

Hence: $\mathcal{E}(t)=\mathop{}\mathopen{}{\large\mathcal{O}}\mathopen{}\left(e^{-\frac{\lambda\gamma}{2}t}\right)$. Since: $F(x(t))-F^{*}=a(t)\leqslant 2\mathcal{E}(t)$ for all $t\geq t_{1}$, we finally get the expected result.

Case where $\theta\in(0,1)$.

Following the strategy proposed by H. Attouch and A. Cabot in the proof of [3, Theorem 3.12] in the unperturbed case, we choose in time-dependent parameters $\lambda$ and $\xi$:

[TABLE]

so that the energy $\mathcal{E}$ can be rewritten as:

[TABLE]

The case $\theta=0$ is excluded in the proof detailed hereafter since, as in [3, Theorem 3.12], we need that: $\lim_{t\rightarrow+\infty}\beta(t)=0$. Extending the proof of [3, Theorem 3.12] to our setting, we obtain the following differential inequality whose proof is detailed in appendix:

Lemma 4.3**.**

Let $\gamma\geq 1$ and $\beta(t)=\frac{\alpha}{t^{\theta}}$. If $F$ satisfies the hypothesis $\mathbf{H}_{1}(\gamma)$, then:

[TABLE]

According to Lemma 4.3 and noticing that for all $t>0$: $\dot{\beta}(t)+\frac{\gamma-2}{\gamma+2}\beta(t)^{2}\leqslant 0$, we then obtain:

[TABLE]

To prove that the energy $\mathcal{G}$ is non increasing and thus bounded, we have now to control the scalar product $\langle x(t)-x^{*},\dot{x}(t)\rangle$. Assuming that $F$ satisfies the growth condition $\mathbf{H}_{2}(2)$ and admits a unique minimizer, we first have:

[TABLE]

where $C=\max(1,\frac{1}{2K_{2}})$. Since $\displaystyle\lim_{t\rightarrow+\infty}\beta(t)=0$, it follows:

[TABLE]

so that there exists $t_{1}\geqslant t_{0}$ such that for all $t\geqslant t_{1}$,

[TABLE]

Combining (4.24), (4.25) and (4.26), we then obtain:

[TABLE]

Observe now that by definition, we have: $\beta(t)^{2}=\mathop{}\mathopen{}{\scriptstyle\mathcal{O}}\mathopen{}\left(\beta(t)\right)$ and $\dot{\beta}(t)=\mathop{}\mathopen{}{\scriptstyle\mathcal{O}}\mathopen{}\left(\beta(t)\right)$. Hence, for any constant $m\in(0,\frac{2\gamma}{\gamma+2})$, there exists $t_{2}\geq t_{1}$ such that for all $t\geqslant t_{2}$:

[TABLE]

or equivalently:

[TABLE]

The rest of the proof is quite standard: before integrating the differential inequality (4.28) between $t_{2}$ and $t$, we first need to control the term $\|\dot{x}(t)+\frac{2}{\gamma+2}\beta(t)(x(t)-x^{*})\|$. To that end, observe that, according to (4.27), the energy $\mathcal{G}$ is non-increasing and that: $\forall t\geqslant t_{2},\leavevmode\nobreak\ \mathcal{G}(t)\leq\mathcal{G}(t_{2}),$ i.e.:

[TABLE]

Moreover, with our choice of parameters, the energy $\mathcal{E}(t)$ is not a sum of non negative terms:

[TABLE]

so that the term $\|\dot{x}(t)+\frac{2\beta(t)}{\gamma+2}(x(t)-x^{*})\|^{2}$ can not be directly controlled by the energy $\mathcal{E}(t)$. But, according to the growth condition $\mathbf{H}_{2}(2)$ combined with the uniqueness of the minimizer, we have:

[TABLE]

for $t$ large enough, and: $\mathcal{E}(t)\geqslant\frac{1}{2}(F(x(t))-F^{*})+\|\dot{x}(t)+\frac{2\beta(t)}{\gamma+2}(x(t)-x^{*})\|^{2}$. Hence:

[TABLE]

where: $c=\sqrt{2\mathcal{E}(t_{2})}$. Applying the Grönwall-Bellman Lemma [14, Lemma A.5], we obtain:

[TABLE]

Assuming that $\int_{t_{2}}^{+\infty}\|g(s)\|ds<+\infty$, we can so conclude that:

[TABLE]

Coming back to (4.28), we obtain the following differential inequality:

[TABLE]

Integrating between $t_{2}$ and $t$ and stating: $\Gamma(t)=\displaystyle\int_{t_{2}}^{t}\beta(s)ds$, we finally obtain: for all $t\geqslant t_{2}$,

[TABLE]

Hence: $\mathcal{E}(t)=\mathop{}\mathopen{}{\large\mathcal{O}}\mathopen{}\left(e^{-m\Gamma(t)}\right)$. Since: $F(x(t))-F^{*}=a(t)\leqslant 2\mathcal{E}(t)$ for all $t\geq t_{2}$, we finally get the expected result and using $\mathbf{H}_{2}(2)$, the other estimates follow directly.

4.3 Proof of Theorem 3.4

The proof of Theorem 3.4 relies on almost the same energy as that used in [10]:

[TABLE]

where $\lambda$ is a non-negative real constant as in [10] and $\xi(.)$ is here a real-valued function. Noting:

[TABLE]

we have: $\mathcal{E}(t)=t(a(t)+b(t)+\xi(t)c(t))$. We also define:

[TABLE]

The proofs of our theorems rely on the following lemma whose proof is detailed in appendix A.4:

Lemma 4.4**.**

Let $\gamma_{1}\geqslant 1$. If $F$ satisfies the hypothesis $\mathbf{H}_{1}(\gamma_{1})$ and if $\xi(t)=\lambda(\lambda+1-\alpha t^{1-\theta})$, then:

[TABLE]

Note $r=\frac{1+\theta}{2}$. Taking $\lambda=\frac{2r}{\gamma_{1}-2}$ and $p=p_{1}+2(r-1)$ with $p_{1}=\frac{4r}{\gamma_{1}-2}$, we obtain:

[TABLE]

Since $r\leqslant 1$, $2\lambda(\lambda+1)(r-1)\leqslant 0$, hence:

[TABLE]

If $\theta<1$, it exists $t_{1}$ depending only on $\gamma_{1}$, $\alpha$ and $\theta$ such that $\mathcal{H}^{\prime}(t)\leqslant 0$ for all $t\geqslant t_{1}$. 2. 2.

If $\theta=1$ and $\alpha\geqslant\frac{\gamma_{1}+2}{\gamma_{1}-2}$, $\mathcal{H}^{\prime}$ is non positive for all $t\geqslant t_{0}$.

We will now use similar reasoning as in [10] to prove the results of our theorem. Since $\mathcal{H}^{\prime}(t)\leqslant 0$, for any choice of $x^{*}$ in the set of minimizers $X^{*}$, the function $\mathcal{H}$ is bounded above and since the set of minimizers is bounded because $F$ is coercive, there exists $A$ > 0 and $t_{0}$ such that for all choices of $x^{*}$ in $X^{*}$:

[TABLE]

Hence for all $x^{*}\in X^{*}$ and $t\geqslant t_{0}$, $\mathcal{H}(t)\leqslant A$. Hence

[TABLE]

We have $\xi(t)=\frac{2r}{\gamma_{1}-2}(\frac{2r}{\gamma_{1}-2}+1-\alpha t^{-2(r-1)})$. Hence:

• •

If $\theta=1$, then $r=1$. Since $\alpha\geqslant\frac{\gamma_{1}+2}{\gamma_{1}-2}\geqslant\frac{\gamma_{1}}{\gamma_{1}-2}$, we have:

[TABLE]

• •

If $\theta\in(0,1)$, then:

[TABLE]

Hence, in both cases:

[TABLE]

Therefore:

[TABLE]

And since this is verified for all $x^{*}\in X^{*}$:

[TABLE]

We set $v(t)=t^{\frac{4r}{\gamma_{2}-2}}d(x(t),X^{*})^{2}$ Then

[TABLE]

Since $F$ satisfies $\mathbf{H}_{2}(\gamma_{2})$, there exists K > 0 such that

[TABLE]

i.e

[TABLE]

Hence

[TABLE]

Back to (4.41), this yields:

[TABLE]

Hence

[TABLE]

Which, since $\gamma_{1}\leqslant\gamma_{2}$, means that $v$ is bounded. Therefore, from (4.41) we deduce that there exists $B>0$ such that:

[TABLE]

Since $\gamma_{1}\leqslant\gamma_{2}$, we have $\frac{-r2\gamma_{2}}{\gamma_{2}-2}\geqslant\frac{-2r\gamma_{1}}{\gamma_{1}-2}$. Hence $F(x(t))-F^{*}=O(t^{-\frac{2r\gamma_{2}}{\gamma_{2}-2}})$.

4.4 Proof of Theorem 3.5

The proof is inspired by the one of Theorem 3.4 where an additional term including the noise is considered.

First, we set parameters $r=\frac{1+\theta}{2}$, $\lambda=\frac{2r}{\gamma_{1}-2}$ and $p=p_{1}+2(r-1)$, with $p_{1}=2\lambda$ and functions $\xi$, $\mathcal{E}$ and $\mathcal{H}$ exactly as in Theorem 3.4. In addition to the energy functions $\mathcal{E}$ and $\mathcal{H}$ we define

[TABLE]

We refer the reader to Appendix (A.4) for a detailed calculation of the following bound on the derivative $\mathcal{G}^{\prime}$

[TABLE]

Actually, the integral term $\mathcal{G}(t)-\mathcal{H}(t)$ is computed such that the bound on $\mathcal{G}^{\prime}(t)$ is equal to the bound (4.33) proposed in Theorem 3.4 for $\mathcal{H}^{\prime}$. Thus we deduce once again that under the hypotheses of the Theorem 3.5, it exists $t_{1}\geqslant 1$ such that the function $\mathcal{G}$ is non increasing for $t\geqslant t_{1}$.

We will need now the following direct lemma

Lemma 4.5**.**

If $F$ satisfies the growth condition $\mathbf{H}_{2}(\gamma_{2})$ with $\gamma_{2}>2$ :

[TABLE]

then defining $p_{2}=\frac{4r}{\gamma_{2}-2}$, with notations defined in (4.3) we have

[TABLE]

It follows that for any $m\in\mathbb{R}$, it exists $M\in\mathbb{R}$ such that for any $t\geqslant t_{0}$

[TABLE]

Since for all $t\geqslant t_{1}$, $\mathcal{G}(t)\leqslant\mathcal{G}(t_{1})$. Then:

[TABLE]

Hence, we have:

[TABLE]

Using the fact that the $|\xi(t)|t^{2(r-1)}$ is uniformly bounded, see (4.38) we get

[TABLE]

Let’s define $p_{2}=\frac{4r}{\gamma_{2}-2}$. Since $\gamma_{1}\leqslant\gamma_{2}$ we have $p_{2}\leqslant p_{1}$ and for any $t\geqslant t_{1}$, $t^{p_{1}-p_{2}}\geqslant 1$. Dividing the previous inequality by $t^{p_{1}-p_{2}}$ we get for any $t\geqslant t_{1}$

[TABLE]

which implies

[TABLE]

Since $F$ satisfies the growth condition $\mathbf{H}(\gamma_{2})$ we can apply Lemma 4.5 and deduce it exists $M\in\mathbb{R}$ such that for $t\geqslant t_{1}$

[TABLE]

which implies using the definition of $b(t)$ given in (4.3)

[TABLE]

Applying the Grönwall Bellman Lemma it follows that

[TABLE]

Under the hypotheses of the Theorem, the right member of the inequality is uniformly bounded relatively to $t$. It follows that it exists $M_{1}\geqslant 0$ such that for any $t\geqslant t_{1}$

[TABLE]

and thus that it exists $M_{2}>0$ such that for any $t\geqslant t_{1}$

[TABLE]

Combining this inequality with (4.57) if follows that for any $t\geqslant t_{1}$

[TABLE]

Using once again Lemma 4.5 we deduce it exists $M_{3}$ and $M_{4}$ such that for any $t\geqslant t_{1}$

[TABLE]

which implies that it exists $M_{5}$ such that any $t\geqslant 1$

[TABLE]

that is

[TABLE]

Which is the desired result.

4.5 Proof of Corollary 3.9

In this paragraph we detail the proof of Corollary 3.9 in which it is stated that, in the flat case, the trajectory of any solution $x$ of the ODE (3.5) is finite.

From the proof of Theorem 3.5 (see (4.61)), there exists $A_{1}>0$ and $t_{1}\geqslant t_{0}$ such that, for all $t\geqslant t_{1}$:

[TABLE]

Combining the growth condition $\mathbf{H}_{2}(\gamma)$ and the conclusion of Theorem 3.5, we have that there exists $t_{2}\geqslant t_{1}$ such that:

[TABLE]

It follows that for all $t\geqslant t_{2}$:

[TABLE]

Hence:

[TABLE]

Noticing that $r-1\leqslant 0$, we finally get:

[TABLE]

Which means that $\|\dot{x}(t)\|$ is integrable and that the trajectory is finite.

Acknowledgement

This study has been carried out with financial support from the French state, managed by the French National Research Agency (ANR GOTMI - ANR-16-VCE33-0010-01) and partially supported by ANR-11-LABX-0040-CIMI within the program ANR-11-IDEX-0002-02.

Appendix A Appendix

A.1 Proof of Lemma 4.1: differentiating the energy function for Theorem 3.2

Differentiating $\mathbf{\mathcal{E}}$.

Consider the energy $\mathcal{E}$ defined as follows by:

[TABLE]

where:

[TABLE]

We then have:

[TABLE]

Since $x$ is a solution of the ODE (3.2), we have:

[TABLE]

Hence:

[TABLE]

Noticing that:

[TABLE]

we deduce that

[TABLE]

Choosing now $\underline{\xi=\lambda(\lambda+1-\alpha)}$, we get:

[TABLE]

Since $F$ satisfies $\mathbf{H_{1}(\gamma)}$:

[TABLE] 2. 2.

Differentiating $\mathcal{H}$ and $\mathcal{G}$

Recall now that:

[TABLE]

where: $\mathcal{H}(t)=t^{p}\mathcal{E}(t)$. Since $\mathcal{E}(t)=t(a(t)+b(t)+\xi c(t))$, we deduce from (A.6) that:

[TABLE]

Hence the expected inequality:

[TABLE]

A.2 Proof of Lemma 4.2: differentiating the energy function of Theorem 3.3

Consider the energy $\mathcal{E}$ defined as follows by:

[TABLE]

where:

[TABLE]

We then have:

[TABLE]

Since $x$ is a solution of the ODE (3.5), we have:

[TABLE]

Hence:

[TABLE]

Noticing that:

[TABLE]

we deduce that

[TABLE]

Choosing now $\underline{\xi=\lambda(\lambda-\alpha)}$, we get:

[TABLE]

Since $F$ satisfies $\mathbf{H}_{1}(\gamma)$, we finally get:

[TABLE]

A.3 Proof of Lemma 4.3: differentiating the energy function of Theorem 3.3 for the classical heavy ball system

Consider the energy:

[TABLE]

where: $\lambda(t)=\frac{2\beta}{\gamma+2}$ and $\xi(t)=-\lambda(t)^{2}$. Then:

[TABLE]

Since $x$ satisfies the ODE (3.5) when $\theta\in(0,1)$, we have:

[TABLE]

so that:

[TABLE]

assuming that $F$ satisfies $\mathbf{H}_{1}(\gamma)$. Noticing that by definition of the energy $\mathcal{E}(t)$:

[TABLE]

we get:

[TABLE]

as expected.

A.4 Proof of Lemma 4.4: differentiating the energy function of Theorem 3.4

Let $\lambda$ be a non-negative real constant and $\xi(.)$ a real-valued function. Consider the energy:

[TABLE]

where:

[TABLE]

Differentiating $\mathcal{E}$

[TABLE]

Since $x$ satisfies the ODE (3.5) with $g\equiv 0$, we have:

[TABLE]

hence:

[TABLE]

Since $F$ satisfies $\mathbf{H}_{1}(\gamma_{1})$, we get:

[TABLE]

Noticing that

[TABLE]

i.e.

[TABLE]

we get:

[TABLE]

Setting $\underline{\xi(t)=\lambda(\lambda+1-\alpha t^{1-\theta})}$, we obtain:

[TABLE] 2. 2.

Differentiating $\mathcal{H}$.

Recall that $\mathcal{H}(t)=t^{p}\mathcal{E}(t)$. Hence $\mathcal{H}^{\prime}(t)=t^{p-1}(p\mathcal{E}(t)+(1+t)\mathcal{E}^{\prime}(t))$ Hence

[TABLE]

as expected.