Accelerated Stochastic Optimization Methods under Quasar-convexity

TL;DR

This paper introduces new stochastic optimization algorithms tailored for quasar-convex functions, achieving faster convergence than existing methods and applicable to problems like learning linear dynamical systems.

Contribution

The paper develops a novel class of stochastic methods specifically designed for smooth quasar-convex functions, with improved convergence rates and a unified analysis framework.

Findings

Algorithms outperform existing methods on several examples

Fast convergence demonstrated for learning linear dynamical systems

Unified analysis for stochastic and deterministic algorithms

Abstract

Non-convex optimization plays a key role in a growing number of machine learning applications. This motivates the identification of specialized structure that enables sharper theoretical analysis. One such identified structure is quasar-convexity, a non-convex generalization of convexity that subsumes convex functions. Existing algorithms for minimizing quasar-convex functions in the stochastic setting have either high complexity or slow convergence, which prompts us to derive a new class of stochastic methods for optimizing smooth quasar-convex functions. We demonstrate that our algorithms have fast convergence and outperform existing algorithms on several examples, including the classical problem of learning linear dynamical systems. We also present a unified analysis of our newly proposed algorithms and a previously studied deterministic algorithm.