Incremental Methods for Weakly Convex Optimization

TL;DR

This paper analyzes incremental methods for weakly convex optimization, establishing their convergence rates and demonstrating their effectiveness through numerical experiments on matrix sensing.

Contribution

It extends convergence rate analysis of incremental methods from convex to weakly convex problems, including new linear convergence results under sharpness conditions.

Findings

Iteration complexity of $O(\varepsilon^{-4})$ for stationarity

Linear convergence under sharpness condition with proper initialization

Numerical validation on robust matrix sensing problem

Abstract

Incremental methods are widely utilized for solving finite-sum optimization problems in machine learning and signal processing. In this paper, we study a family of incremental methods -- including incremental subgradient, incremental proximal point, and incremental prox-linear methods -- for solving weakly convex optimization problems. Such a problem class covers many nonsmooth nonconvex instances that arise in engineering fields. We show that the three said incremental methods have an iteration complexity of $O(\varepsilon^{-4})$ for driving a natural stationarity measure to below $\varepsilon$. Moreover, we show that if the weakly convex function satisfies a sharpness condition, then all three incremental methods, when properly initialized and equipped with geometrically diminishing stepsizes, can achieve a local linear rate of convergence. Our work is the first to extend the convergence rate analysis of incremental methods from the nonsmooth convex regime to the weakly convex regime. Lastly, we conduct numerical experiments on the robust matrix sensing problem to illustrate the convergence performance of the three incremental methods.