Smoothness parameter of power of Euclidean norm
Anton Rodomanov, Yurii Nesterov

TL;DR
This paper investigates the derivatives of Euclidean norm powers, proving their Hölder continuity, deriving explicit constants, and optimizing these constants for odd derivatives and integer powers.
Contribution
It provides explicit expressions and optimal constants for the Hölder continuity of derivatives of Euclidean norm powers, improving understanding of their smoothness properties.
Findings
Constants are optimal for odd derivatives.
Constants are at most twice suboptimal for even derivatives.
Improved results for integer powers with Lipschitz continuity.
Abstract
In this paper, we study derivatives of powers of Euclidean norm. We prove their H\"older continuity and establish explicit expressions for the corresponding constants. We show that these constants are optimal for odd derivatives and at most two times suboptimal for the even ones. In the particular case of integer powers, when the H\"older continuity transforms into the Lipschitz continuity, we improve this result and obtain the optimal constants.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsOptimization and Variational Analysis · Mathematical Inequalities and Applications · Advanced Optimization Algorithms Research
∎
11institutetext: Anton Rodomanov 22institutetext: ICTEAM, Catholic University of Louvain, Louvain-la-Neuve, Belgium
[email protected] 33institutetext: Yurii Nesterov 44institutetext: Center for Operations Research and Economics, Catholic University of Louvain, Louvain-la-Neuve, Belgium
Smoothness Parameter of Power of Euclidean Norm
††thanks: Communicated by Liqun Qi.
Anton Rodomanov
Yurii Nesterov
(Received: 29 July 2019 / Accepted: 6 March 2020 / Published online: 27 March 2020
© The Author(s) 2020 )
Abstract
In this paper, we study derivatives of powers of Euclidean norm. We prove their Hölder continuity and establish explicit expressions for the corresponding constants. We show that these constants are optimal for odd derivatives and at most two times suboptimal for the even ones. In the particular case of integer powers, when the Hölder continuity transforms into the Lipschitz continuity, we improve this result and obtain the optimal constants.
Keywords:
Hölder continuity polynomials optimal constants
MSC:
26A16 46G05 11C08
††journal: JOTA
1 Introduction
Starting from the paper nesterov2006cubic , there has been an increasing interest in the cubic regularization of Newton’s method (see, for example, nesterov2008accelerating ; cartis2011adaptive1 ; cartis2011adaptive2 ; carmon2016gradient ; kohler2017sub ; cartis2018global ; doikov2018randomized ), which has some attractive global worst-case complexity guarantees. The main idea of this method is to approximate the objective function with its second-order Taylor approximation, add to it the cube of Euclidean norm with certain coefficient and then minimize the result to obtain a new point.
A natural generalization of this approach consists in considering a general high-order Taylor approximation together with a certain high-order power of Euclidean norm as a regularizer. This leads to tensor methods schnabel1991tensor ; baes2009estimate ; cartis2017improved ; gasnikov2019near , that have recently gained their popularity after it was shown in nesterov2015implementable that one step of the third-order tensor method for minimizing convex functions is comparable with that of the cubic Newton method.
For some applications, involving functions with Hölder continuous derivatives, it may also be reasonable to regularize the models with fractional degrees of the Euclidean norm, as discussed in grapiglia2017regularized and grapiglia2019tensor .
The efficiency of all the aforementioned methods strongly depends on our possibilities in solving the corresponding auxiliary problems, that arise at each iteration. Therefore, it is important to be able to quickly solve minimization problems regularized by powers of Euclidean norm.
Two of the most important characteristics of the objective function, that influence the convergence rate of minimization algorithms, are the constants of uniform convexity and Hölder continuity of derivatives. It is thus important to know these parameters for powers of Euclidean norm in order to justify the convergence rates of the related minimization algorithms.
The uniform convexity of powers of Euclidean norm was first investigated in vladimirov1978uniformly , where the authors obtained optimal constants for all integer powers. This result was then generalized to arbitrary real powers in (doikov2019minimizing, , Lemma 5). Thus, the question of uniform convexity is completely solved.
The question of the Hölder continuity of derivatives of powers of Euclidean norm is more subtle. There exist only partial results for some special powers. For example, for any real power between one and two, the Hölder continuity of the first derivative follows from the duality between uniform convexity and Hölder smoothness (see (nesterov2015universal, , Lemma 1)). For any real power between two and three, the Hölder continuity of the second derivative has recently been proved in (doikov2019minimizing, , Example 2), where some suboptimal constants have been obtained. However, there are currently no general results for an arbitrary power.
Thus, establishing Hölder continuity of derivatives of powers of Euclidean norm and estimating the corresponding constants is still an open problem and constitutes the main topic of this work.
This paper is organized as follows. In Section 2, we introduce notation and recall important facts on the norm of symmetric multilinear operators.
In Section 3, we derive a general formula for derivatives of powers of Euclidean norm (Theorem 3.1). The main object in this formula is a certain family of recursively defined polynomials (Definition 1). We give the corresponding definition and provide several examples.
In Sections 4 and 5, we study these polynomials in more detail. We establish useful identities and prove several important properties such as symmetry (Proposition 1), non-negativity (Proposition 3) and monotonicity (Proposition 4). Section 5 is devoted to estimating the Hölder constants of the polynomials. The main results in this section are Theorem 5.1 and Theorem 5.2.
In Section 6, we apply the auxiliary results obtained in the previous sections for proving Hölder continuity of derivatives of powers of Euclidean norm. Namely, in Theorem 6.1, we derive a lower bound for the possible values of Hölder constants. In Theorem 6.2, we prove Hölder continuity of the derivatives along the lines passing through the origin. Finally, in Theorem 6.3, we extend this result onto the whole space and discuss the optimality of the constants.
In the final Section 7, we show how to improve our general result for integer powers, when the Hölder condition corresponds to the Lipschitz condition.
2 Notation and Generalities
In this text, is a finite dimensional real vector space. Its dual space, composed of all linear functionals on , is denoted by . The value of a linear functional , evaluated at a point , is denoted by . To introduce a Euclidean norm on , we fix a self-adjoint positive definite operator and define .
For a function , defined on an open set in , and for an integer , the -th derivative of , if exists, is denoted by . This derivative is a mapping from to the space of symmetric -multilinear forms on .
Let be a -multilinear form on . Its value, evaluated at , is denoted by . When for some , we abbreviate this as . The norm of is defined in the standard way:
[TABLE]
If the form is symmetric, it is known that the maximum in the above definition can be achieved when all the vectors are the same:
[TABLE]
(see, for example, Appendix 1 in nesterov1994interior ).
For , by we denote the -th power of the Euclidean norm:
[TABLE]
The main goal of this paper is to establish that, for any integer and any real , the -th derivative of is -Hölder continuous:
[TABLE]
for all , where is an explicit constant dependent on and .
3 Derivatives of Powers of Euclidean Norm
We start with deriving a general formula for derivatives of the function . The main objects in this formula are univariate polynomials, defined below.
Definition 1
For each integer and each , we define a polynomial as follows. When , we set . For all other ,
[TABLE]
Each polynomial is a combination of the previous polynomial and its derivative . The first five polynomials can be written explicitly:
[TABLE]
Let us now describe how derivatives of are related to polynomials .
Theorem 3.1
For any real , the function is times differentiable for all integer . The corresponding derivatives are
[TABLE]
where is an arbitrary unit vector and
[TABLE]
Proof
Note that is infinitely differentiable on since its restriction on this set is a composition of two infinitely differentiable functions, namely the quadratic function and the power function . Hence, we only need to prove that is also times differentiable at the origin for any , and that (2) holds.
We proceed by induction. The case is trivial since, by definition, the zeroth derivative of a function is the function itself, while for any . Let us assume that , and the claim is proved for .
First, let us justify (2) for any . By the induction hypothesis,
[TABLE]
for all . Differentiating, we obtain that
[TABLE]
for all , and hence
[TABLE]
where the last equality follows from Definition 1.
Now let us show that is also times differentiable at the origin with (this is what (2) says when ). By our inductive assumption, we already know that . Therefore, according to the definition of derivative, it remains to show that , or, equivalently, in view of (1), that
[TABLE]
Applying our inductive assumption, we obtain that
[TABLE]
for all . Since , we have as . Thus, we need to show that is uniformly bounded for all and all unit . Indeed, by Cauchy-Schwartz inequality, we have . Hence . The right-hand side in the above inequality is finite, since a continuous function always achieves its maximum on a compact interval.∎
4 Main Properties of Polynomials
Let us study the polynomials introduced in Definition 1. Our first observation is that , as a function, is always either even or odd.
Proposition 1
For any integer , and any , has the same parity as , i.e. for all .
Proof
Easily follows from Definition 1 by induction.∎
Next we establish identities with the first and second derivatives of .
Lemma 1
For any integer , and any ,
[TABLE]
Proof
Follows from Definition 1 using standard rules of differentiation.∎
Lemma 2
For any integer , and any ,
[TABLE]
Proof
We proceed by induction on . For , by Definition 1, we have while and , so the claim is obviously true. Now let us prove the claim for , assuming that it is already true for all integer . By Definition 1, we have
[TABLE]
Rearranging, we obtain
[TABLE]
By the induction hypothesis, applied for , we have
[TABLE]
for all . Differentiating both sides, we obtain from this that
[TABLE]
Combining the above three formulas, we see that
[TABLE]
At the same time, by Lemma 1, we have
[TABLE]
and, by Definition 1, we also have
[TABLE]
Summing the above two identities, we obtain the right-hand side of (6).∎
Lemma 3
For any integer , and any ,
[TABLE]
Proof
Apply Lemma 2 to the last term in (5).∎
The following lemma is particularly interesting. It turns out that, up to a constant factor, the derivative of the polynomial is exactly the previous polynomial but with a shifted value of .
Lemma 4
For any integer , and any , we have .
Proof
We proceed by induction on . Let . For , we know from Definition 1 that , while , therefore the claim is indeed true. Now let us prove the claim for assuming that it is already proved for all integer . From Lemma 3, we already know that
[TABLE]
Therefore it remains to prove that
[TABLE]
By the induction hypothesis for , we already have the identity and in particular . Thus,
[TABLE]
It remains to verify that
[TABLE]
But this is given directly by Definition 1.∎
Combined with Definition 1, Lemma 4 gives us a useful recursive formula for , that does not involve any derivatives.
Lemma 5
For any integer , and any ,
[TABLE]
Lemma 5 has several corollaries. The first one gives us closed-form expressions for the values of at the boundary points of the interval .
Proposition 2
For any integer , and any , we have111For a positive integer , by we denote the double factorial of (the product of all integers between 1 and with the same parity as ). We also define .
[TABLE]
and
[TABLE]
Proof
We proceed by induction on . From Definition 1, we have and , . Thus, the claim is indeed true for and . Now let us prove the claim for assuming that it is already true for all integer . Using Lemma 5, we obtain
[TABLE]
By the induction hypothesis, applied for (and ), we have
[TABLE]
By shifting the index in the product, this can be rewritten as
[TABLE]
Substituting this into (10), we obtain (8).
Similarly, by Lemma 5, we also have . But by the induction hypothesis, , and we obtain (9).∎
The second corollary of Lemma 5 states that cannot take negative values on the interval , provided that is sufficiently large.
Proposition 3
For any integer , and any real , is non-negative on .
Proof
We proceed by induction on . Let . For , we know, by Definition 1, that , which is actually non-negative for all real . For , by Definition 1, we have , which is indeed non-negative when .
Now let us prove the claim for , assuming that it is already proved for all integer . From Lemma 5, we know that
[TABLE]
By the induction hypothesis, applied respectively for , and , (observe that in both cases since ), we have and . Since , then also , and . Thus, all parts in the right-hand side of the above formula are non-negative.∎
Combining Proposition 3 with Lemma 4, we obtain that, when , the polynomial is not only non-negative but also monotonically increasing.
Proposition 4
For any integer , and any real , the derivative is non-negative on ; hence is monotonically increasing on .
Finally, let us show how we can apply the properties, that we have established above, to finding the maximal absolute value of on .
Proposition 5
For any integer , and any real ,
[TABLE]
Proof
By Proposition 1, we have . Since is non-negative on (Proposition 3), . By Proposition 4, . But according to Proposition 2.∎
5 Hölder Constants of Polynomials
We continue our study of polynomials , but now we restrict our attention to the particular case when for some real .
Clearly, the polynomial is -Hölder continuous on , since this is true for any other polynomial on a compact interval. The goal of this section is to obtain an explicit expression for the corresponding Hölder constant. We start with the result, allowing us to reduce our task to that on .
Theorem 5.1
For any integer , and any real , the polynomial is -Hölder continuous on with constant
[TABLE]
where is the corresponding Hölder constant of on .
Proof
Let . We need to prove that
[TABLE]
By Proposition 1, this inequality is invariant to negation transformations . Therefore, we can assume that . Furthermore, we can assume that , since otherwise the claim is trivial.
Case I. Suppose is even. Then, by Proposition 1,
[TABLE]
Note that . Therefore, by Hölder condition on ,
[TABLE]
At the same time, by the triangle inequality, and (11) follows.
Case II. Now suppose is odd. By Proposition 1 and Proposition 3,
[TABLE]
Recall that (Proposition 2). Therefore,
[TABLE]
Hence,
[TABLE]
To prove (11), it remains to show that . But this follows from the concavity of power function .∎
Our next task is to estimate the Hölder constant of on :
[TABLE]
Note that Proposition 4 allows us to remove the absolute value sign.
Theorem 5.2
For any integer , and any real , we have
[TABLE]
The proof of Theorem 5.2 is based on two auxiliary propositions.
Proposition 6
For any integer and any real , the function
[TABLE]
is monotonically increasing on .
Proposition 7
For any integer and any real , the function
[TABLE]
is monotonically decreasing on .
Let us assume for a moment that these propositions are already proved. Then, the proof of Theorem 5.2 is simple.
Proof
Let . From Proposition 6, we know that
[TABLE]
Therefore, to prove (13), it remains to show that
[TABLE]
Recall that, by Proposition 2, we have . Thus, the inequality we need to prove is
[TABLE]
or, equivalently,
[TABLE]
But this follows from Proposition 7.
Our goal now is to prove Proposition 6 and Proposition 7.
We start with Proposition 6. It requires three technical lemmas.
Lemma 6
For any integer , and any real ,
[TABLE]
Moreover, when ,
[TABLE]
Proof
First, let us prove (17). By Lemma 1, we have
[TABLE]
Since (Proposition 4) and , it follows that
[TABLE]
At the same time, by Definition 1,
[TABLE]
Thus,
[TABLE]
Applying Lemma 4, we obtain that
[TABLE]
and (17) follows.
It remains to prove (16). For , we have (Definition 1), hence , and (16) is indeed true. For , by Definition 1, we have , hence , and (16) is again true. The general case easily follows from (17) by induction.∎
Lemma 7
For any integer , any real , and ,
[TABLE]
Proof
We use induction in . For , we have , while (see Definition 1), so the claim is true. For , we have while , (see Definition 1), hence the claim is again true.
Now we prove the claim for assuming that it is already true for all integer . According to Lemma 5, we have
[TABLE]
Since (Proposition 3), we further have
[TABLE]
If , it follows that , and the proof in this case is finished, because the right-hand side in (18) is always non-negative in view of Lemma 6. Therefore, we can assume that .
Since , then
[TABLE]
Applying the inductive assumption to , we obtain
[TABLE]
Hence,
[TABLE]
Thus, to finish the proof, it remains to show that
[TABLE]
But this is guaranteed by Lemma 6.∎
Lemma 8
For any integer , and any real , the function
[TABLE]
is monotonically decreasing on .
Proof
The function (19) is differentiable with derivative
[TABLE]
which is non-positive on by Lemma 7.∎
Now we can present the proof of Proposition 6:
Proof
Since (14) is differentiable, it suffices to prove that its derivative
[TABLE]
is non-negative for all , or, equivalently, that
[TABLE]
By Lemma 2,
[TABLE]
Therefore, it is enough to prove that
[TABLE]
or, equivalently,
[TABLE]
But this immediately follows from Lemma 8 using (20).∎
It remains to prove Proposition 7. For this, we need one more lemma.
Lemma 9
For any integer , and any real , we have
[TABLE]
Proof
As usual, we use induction in . The base case is trivial, since , while (see Definition 1). To prove the general case , we assume that (22) is already true for all integer .
Our first step is to show that
[TABLE]
If , we have , while and (see Definition 1), so (23) is indeed true. To justify it for all other , we proceed as follows. By Lemma 5, we know that
[TABLE]
Therefore, (23) is equivalent to
[TABLE]
By our inductive assumption (22), applied to , we already have
[TABLE]
At the same time, by Lemma 4,
[TABLE]
Thus, (23) is established.
Now we estimate the right-hand side in (23). Applying Lemma 3 and the fact that (Proposition 4), we obtain
[TABLE]
From this, it follows that
[TABLE]
Substituting the above equation into (23), we obtain
[TABLE]
Since (by Proposition 3), it only remains to show that . But this follows from the concavity of power function .∎
Now we can give the proof of Proposition 7:
Proof
Since (15) is differentiable, it suffices to prove that its derivative
[TABLE]
is non-positive for all . By Lemma 2, we have
[TABLE]
Thus, we need to show that
[TABLE]
or, equivalently (by multiplying both sides by ), that
[TABLE]
or, equivalently (by moving the first term into the right-hand side), that
[TABLE]
But this is given by Lemma 9.∎
To conclude this section, let us discuss the optimality of Theorem 5.2.
For odd values of , the obtained constant turns out to be optimal. Indeed, using , in (12) and taking into account Proposition 2, we obtain that
[TABLE]
However, for even , this constant is suboptimal. For example, consider the case when . We know that
[TABLE]
The corresponding optimal constant, according to Proposition 6, is
[TABLE]
Note that this maximization problem is logarithmically concave in . Taking the logarithm and setting the derivative to zero, we find that the maximal point corresponds to , and the corresponding optimal value is
[TABLE]
Of course, the last inequality is strict for all .
6 Hölder Continuity of Derivatives of Powers of Euclidean Norm
We have established the main properties of polynomials and obtained an explicit upper bound on their Hölder constant. Hence, we are ready to prove the Hölder continuity of derivatives of powers of Euclidean norm. Let us start with a simple result, that gives us a lower bound on the Hölder constant.
Theorem 6.1
For any integer , and any real , the Hölder constant of , corresponding to degree , cannot be smaller than
[TABLE]
Proof
According to (1), we need to show that
[TABLE]
for some and some unit . Let us choose an arbitrary unit vector , and set . By Theorem 3.1 and Proposition 2,
[TABLE]
To specify , we consider two cases.
If is even, set . Then, by Theorem 3.1, and
[TABLE]
which is exactly . If is odd, we take . This gives us
[TABLE]
where we apply Proposition 1 to rewrite . Hence,
[TABLE]
which is again precisely .∎
Next we prove Hölder continuity with the optimal constant along any line, passing through the origin.
Theorem 6.2
For any integer , and any real , the restriction of to a line, passing through the origin, is -Hölder continuous with constant .
Proof
Let be arbitrary points, lying on a line, passing through the origin, and let be an arbitrary unit vector. According to (1) and Theorem 3.1, we need to show that
[TABLE]
where , .
Observe that this inequality is symmetric in and and is invariant when we replace the pair with . Therefore, we can assume that and .
Since and lie on a line, passing through the origin, and can differ only in sign. Hence, by Proposition 1, we have two options: either or .
Case I. Suppose (while can be of any sign). Then,
[TABLE]
By Proposition 5 and (24), we know that
[TABLE]
Thus, it suffices to prove that . But this follows from the well-known inequality (which is valid for any real ) combined with the reverse triangle inequality.
Case II. Suppose (). By Proposition 1 and Proposition 3, this happens only if is odd and . Thus , and
[TABLE]
Due to (25), it remains to prove that . But this is immediate. Indeed, by the concavity of the power function , while since the segment contains the origin..∎
Our final step is to extend Hölder continuity from lines, passing through the origin, onto the whole space. The main instrument for doing this is exploiting Hölder continuity of , that we studied in Section 5.
Theorem 6.3
For any integer , and any real , is -Hölder continuous with constant
[TABLE]
where is the constant of -Hölder continuity of on . In particular, is -Hölder continuous with constant
[TABLE]
Proof
First of all, observe that the constant is not smaller than the corresponding lower bound given by Theorem 6.1:
[TABLE]
Indeed, for odd values of , these constants coincide. When is even, (28) follows from the following trivial lower bound for the Hölder constant :
[TABLE]
where the last equality is due to Proposition 2.
Secondly, observe that we only need to prove the first claim, since the other one follows directly from the first one and Theorem 5.2.
Let , and let be an arbitrary unit vector. In view of (1) and Theorem 3.1, we need to show that
[TABLE]
where , .
Due to invariance of the above inequality to transformations of the form and , we can assume in what follows, that and . Furthermore, we can also assume that (and hence ), since otherwise the claim follows from Theorem 6.2.
There are now several cases to consider.
Case I. Suppose . By Propositions 1 and 3, this happens only if is odd and . Then, , and
[TABLE]
where we have removed the absolute value sign, because all terms in the right-hand side are non-negative (see Proposition 3).
Since is odd, (see Proposition 1). Therefore, by the definition of , it follows that
[TABLE]
Combining this with the concavity of power function , we obtain
[TABLE]
Note that by Theorem 5.2. Thus, it remains to show that . But this follows from the Cauchy-Schwartz inequality since by the definition of , .
Case II. Now suppose (while can have any sign). We prove (29) by proving separately two inequalities with the removed absolute value sign.
First, we show that
[TABLE]
Let be the radial projection of onto the sphere with radius , centered at the origin. Note that
[TABLE]
The first two relations are evident. The last one follows from the fact that projections onto convex sets decrease distances, and can be explicitly proved as follows. First, by the Cauchy-Schwartz inequality, we have , where . Therefore, using the fact that , we obtain
[TABLE]
Since (Proposition 3), from (31) it follows that
[TABLE]
At the same time, by Theorem 5.1,
[TABLE]
Hence,
[TABLE]
Note that
[TABLE]
Therefore, by Cauchy-Schwartz inequality and (31), we have
[TABLE]
Substituting this into (32), we obtain
[TABLE]
This finishes the proof of (30), because by Theorem 5.2.
It remains to show the reverse inequality
[TABLE]
For this, we consider two subcases.
Case II(a). Suppose . Let be the projection of onto the line, connecting with the origin, and let . Then,
[TABLE]
Furthermore,
[TABLE]
Indeed, if , then and ; otherwise , and hence (Proposition 1), which either coincides with when is even, or becomes when is odd (see Proposition 3).
Since (Proposition 4), it follows from (35) that . Using also (34) and the fact that , we obtain . Thus,
[TABLE]
Note that in the right-hand side, we have the difference of the derivatives and , where the points and lie on a line, passing through the origin. Therefore, from Theorem 6.2, it follows that
[TABLE]
which proves (33), in view of (28) and (34).
Case II(b). Now suppose . Denote by the constant of -Hölder continuity of the polynomial on the interval . To prove (33), it suffices to show that
[TABLE]
Indeed, recall that . Also
[TABLE]
by the Cauchy-Schwartz inequality. Therefore, if (36) is true, then
[TABLE]
where in view of Proposition 2, Theorem 5.1 and Theorem 5.2. Thus, it remains to show (36), or, equivalently, that
[TABLE]
Denote . We need to prove that
[TABLE]
Note that the right-hand side of this inequality, as a function of , is concave (and well-defined, since ). Hence, to justify (37), we only need to prove the following two boundary cases:
[TABLE]
But both of them follow from the definition of .∎
Comparing the result of Theorem 6.3 with the lower bound , given by Theorem 6.1, we see, that for odd values of , the constant is optimal. Unfortunately, this is no longer true for even values of . Nevertheless, the constant is still quite accurate. Indeed, since
[TABLE]
we have
[TABLE]
Thus, the constant is at most two times suboptimal: .
One may think that the reason, why we obtained a suboptimal bound for even values of , is related to the fact that we had used a suboptimal value for the Hölder constant of the polynomial (see the corresponding discussion at the end of Section 5). However, this is not the actual reason. Indeed, let us look what happens when we use the optimal value for in the particular case . Recall that the optimal constant in this case is
[TABLE]
Substituting this expression into (27), we obtain an improved estimate
[TABLE]
However, this new estimate is still different from the lower bound
[TABLE]
At the same time, for small values of , the difference between and is almost negligible.
7 Lipschitz Constants of Derivatives of Powers of Euclidean Norm
For even values of , our estimate of the Hölder constant of was suboptimal. It turns out, that in the special case when , it is actually very simple to eliminate this drawback and obtain an optimal constant for all values of . This case corresponds to Lipschitz continuity.
Theorem 7.1
For any integer , the derivative is Lipschitz continuous with constant
[TABLE]
where for a non-negative integer denotes the factorial of .
Proof
It suffices to prove that for all and all unit . By Theorem 3.1, we have . Since , we obtain . The claim now follows from Proposition 5.∎
8 Conclusions
In this work, we have proved that derivatives of powers of Euclidean norm are Hölder continuous and have obtained explicit expressions for the corresponding Hölder constants. We have shown that our constants are optimal for odd derivatives and at most two times suboptimal for the even ones. In the particular case of integer powers, when the Hölder condition corresponds to the Lipschitz condition, we have managed to improve our result and obtained optimal constants in all cases. We believe that in general it should be possible to obtain optimal constants for even derivatives as well. However, this seems to be a difficult problem.
Acknowledgements.
The presented results were supported by ERC Advanced Grant 788368. We are grateful to the anonymous referees for their attentive reading and valuable comments.
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1(1) Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108 (1), 177–205 (2006)
- 2(2) Nesterov, Y.: Accelerating the cubic regularization of Newton’s method on convex problems. Mathematical Programming 112 (1), 159–181 (2008)
- 3(3) Cartis, C., Gould, N.I., Toint, P.L.: Adaptive cubic regularisation methods for unconstrained optimization. Part I: Motivation, convergence and numerical results. Mathematical Programming 127 (2), 245–295 (2011)
- 4(4) Cartis, C., Gould, N.I., Toint, P.L.: Adaptive cubic regularisation methods for unconstrained optimization. Part II: Worst-case function-and derivative-evaluation complexity. Mathematical programming 130 (2), 295–319 (2011)
- 5(5) Carmon, Y., Duchi, J.C.: Gradient descent efficiently finds the cubic-regularized non-convex Newton step. ar Xiv preprint ar Xiv:1612.00547 (2016)
- 6(6) Kohler, J.M., Lucchi, A.: Sub-sampled cubic regularization for non-convex optimization. In: Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 1895–1904. JMLR (2017)
- 7(7) Cartis, C., Scheinberg, K.: Global convergence rate analysis of unconstrained optimization methods based on probabilistic models. Mathematical Programming 169 (2), 337–375 (2018)
- 8(8) Doikov, N., Richtarik, P., et al.: Randomized block cubic Newton method. In: International Conference on Machine Learning, pp. 1289–1297 (2018)
