Optimal Stochastic Non-smooth Non-convex Optimization through Online-to-Non-convex Conversion

TL;DR

This paper introduces new algorithms for non-smooth, non-convex stochastic optimization that improve complexity bounds by leveraging a reduction to online learning, achieving optimal or near-optimal results.

Contribution

The paper presents a novel reduction technique from non-smooth non-convex optimization to online learning, leading to improved complexity bounds and unifying existing results.

Findings

Reduced complexity for finding $(oldsymbol{ ext{δ}},oldsymbol{ ext{ε}})$-stationary points to $O( ext{ε}^{-3} ext{δ}^{-1})$

Achieved a complexity of $O( ext{ε}^{-1.5} ext{δ}^{-0.5})$ for smooth objectives using optimistic online learning

Unified and recovered optimal results for smooth and second-order smooth objectives in stochastic and deterministic settings.

Abstract

We present new algorithms for optimizing non-smooth, non-convex stochastic objectives based on a novel analysis technique. This improves the current best-known complexity for finding a $(\delta,\epsilon)$-stationary point from $O(\epsilon^{-4}\delta^{-1})$ stochastic gradient queries to $O(\epsilon^{-3}\delta^{-1})$, which we also show to be optimal. Our primary technique is a reduction from non-smooth non-convex optimization to online learning, after which our results follow from standard regret bounds in online learning. For deterministic and second-order smooth objectives, applying more advanced optimistic online learning techniques enables a new complexity of $O(\epsilon^{-1.5}\delta^{-0.5})$. Our techniques also recover all optimal or best-known results for finding $\epsilon$ stationary points of smooth or second-order smooth objectives in both stochastic and deterministic settings.