Continuized Acceleration for Quasar Convex Functions in Non-Convex Optimization

TL;DR

This paper demonstrates that continuized Nesterov acceleration can optimally minimize quasar convex functions, broadening its applicability to training generalized linear models and certain non-convex functions.

Contribution

It applies continuized Nesterov acceleration to quasar convex functions, achieving optimal bounds and extending applicability to GLMs and other non-convex functions.

Findings

Achieves optimal gradient evaluation bounds for quasar convex functions.

Shows GLM training objectives satisfy quasar convexity.

Identifies conditions for accelerated linear rates in specific non-convex functions.

Abstract

Quasar convexity is a condition that allows some first-order methods to efficiently minimize a function even when the optimization landscape is non-convex. Previous works develop near-optimal accelerated algorithms for minimizing this class of functions, however, they require a subroutine of binary search which results in multiple calls to gradient evaluations in each iteration, and consequently the total number of gradient evaluations does not match a known lower bound. In this work, we show that a recently proposed continuized Nesterov acceleration can be applied to minimizing quasar convex functions and achieves the optimal bound with a high probability. Furthermore, we find that the objective functions of training generalized linear models (GLMs) satisfy quasar convexity, which broadens the applicability of the relevant algorithms, while known practical examples of quasar convexity in non-convex learning are sparse in the literature. We also show that if a smooth and one-point strongly convex, Polyak-Lojasiewicz, or quadratic-growth function satisfies quasar convexity, then attaining an accelerated linear rate for minimizing the function is possible under certain conditions, while acceleration is not known in general for these classes of functions.