Complexity of Projected Gradient Methods for Strongly Convex Optimization with H\"older Continuous Gradient Terms

TL;DR

This paper analyzes the complexity of projected gradient methods for strongly convex optimization problems with component functions having H"older continuous gradients, extending existing results to cases with non-globally H"older continuous gradients.

Contribution

It provides new complexity bounds for projected gradient methods under H"older continuity, including fixed and adaptive stepsize schemes, for a broader class of functions.

Findings

Complexity depends on the minimum H"older exponent among components.

Fixed stepsize yields a certain complexity bound, while adaptive stepsize improves it.

Numerical examples demonstrate the theoretical results in elliptic equations with non-Lipschitz terms.

Abstract

This paper studies the complexity of projected gradient descent methods for a class of strongly convex constrained optimization problems where the objective function is expressed as a summation of $m$ component functions, each possessing a gradient that is H\"older continuous with an exponent $\alpha_i \in (0, 1]$. Under this formulation, the gradient of the objective function may fail to be globally H\"older continuous, thereby rendering existing complexity results inapplicable to this class of problems. Our theoretical analysis reveals that, in this setting, the complexity of projected gradient methods is determined by $\hat{\alpha} = \min_{i \in \{1, \dotsc, m\}} \alpha_i$. We first prove that, with an appropriately fixed stepsize, the complexity bound for finding an approximate minimizer with a distance to the true minimizer less than $\varepsilon$ is $O (\log (\varepsilon^{-1}) \varepsilon^{2 (\hat{\alpha} - 1) / (1 + \hat{\alpha})})$, which extends the well-known complexity result for $\hat{\alpha} = 1$. Next we show that the complexity bound can be improved to $O (\log (\varepsilon^{-1}) \varepsilon^{2 (\hat{\alpha} - 1) / (1 + 3 \hat{\alpha})})$ if the stepsize is updated by the universal scheme. We illustrate our complexity results by numerical examples arising from elliptic equations with a non-Lipschitz term.