On The Convergence of First Order Methods for Quasar-Convex Optimization

TL;DR

This paper investigates the convergence of first-order optimization methods for quasar-convex functions, showing they can achieve complexity bounds similar to convex functions, thus bridging the gap between theory and deep learning practice.

Contribution

It introduces convergence analysis for first-order methods on quasar-convex functions, providing complexity bounds comparable to convex optimization and better than general non-convex results.

Findings

First-order methods converge efficiently on quasar-convex functions.

Complexity bounds are similar to those for convex functions.

Results outperform existing rates for non-convex functions.

Abstract

In recent years, the success of deep learning has inspired many researchers to study the optimization of general smooth non-convex functions. However, recent works have established pessimistic worst-case complexities for this class functions, which is in stark contrast with their superior performance in real-world applications (e.g. training deep neural networks). On the other hand, it is found that many popular non-convex optimization problems enjoy certain structured properties which bear some similarities to convexity. In this paper, we study the class of \textit{quasar-convex functions} to close the gap between theory and practice. We study the convergence of first order methods in a variety of different settings and under different optimality criterions. We prove complexity upper bounds that are similar to standard results established for convex functions and much better that state-of-the-art convergence rates of non-convex functions. Overall, this paper suggests that \textit{quasar-convexity} allows efficient optimization procedures, and we are looking forward to seeing more problems that demonstrate similar properties in practice.