Revisiting the Geometrically Decaying Step Size: Linear Convergence for Smooth or Non-Smooth Functions

TL;DR

This paper demonstrates that a geometrically decaying step size guarantees linear convergence for locally Lipschitz functions, even without convexity or related assumptions, using a simple subgradient method.

Contribution

It introduces a minimal-knowledge subgradient descent algorithm that achieves linear convergence under weaker conditions than traditional assumptions.

Findings

Linear convergence for non-convex functions with geometrically decaying step size

Applicable to both smooth and non-smooth functions

Requires minimal problem-specific knowledge

Abstract

We revisit the geometrically decaying step size given a positive inverse condition number, under which a locally Lipschitz function shows linear convergence. The positivity does not require the function to satisfy convexity, weak convexity, quasar convexity, or sharpness, but instead amounts to a property strictly weaker than the assumptions used in existing works (e.g., weak convexity + sharpness). We propose a clean and simple subgradient descent algorithm that requires minimal knowledge of problem constants, applicable to either smooth or non-smooth functions.