Stochastic Proximal Methods for Non-Smooth Non-Convex Constrained Sparse Optimization

TL;DR

This paper introduces stochastic proximal gradient methods for non-smooth, non-convex constrained optimization, providing the first non-asymptotic convergence analysis and demonstrating superior empirical performance over deterministic algorithms.

Contribution

It presents the first non-asymptotic convergence results for stochastic proximal methods in non-smooth, non-convex constrained settings and introduces algorithms with improved convergence complexities.

Findings

Algorithms outperform state-of-the-art deterministic methods in experiments.

First non-asymptotic convergence results for this class of problems.

Algorithms show superior convergence in numerical tests.

Abstract

This paper focuses on stochastic proximal gradient methods for optimizing a smooth non-convex loss function with a non-smooth non-convex regularizer and convex constraints. To the best of our knowledge we present the first non-asymptotic convergence results for this class of problem. We present two simple stochastic proximal gradient algorithms, for general stochastic and finite-sum optimization problems, which have the same or superior convergence complexities compared to the current best results for the unconstrained problem setting. In a numerical experiment we compare our algorithms with the current state-of-the-art deterministic algorithm and find our algorithms to exhibit superior convergence.