The Stochastic Multi-Proximal Method for Nonsmooth Optimization

TL;DR

This paper introduces the Stochastic Multi-Proximal Method (SMPM), a versatile stochastic algorithm for nonsmooth optimization that generalizes existing methods and achieves fast convergence, with applications to distributed optimization.

Contribution

The paper proposes SMPM, a novel stochastic variance-reduced algorithm for nonsmooth optimization, unifying and extending several existing methods with new convergence guarantees.

Findings

Linear convergence under strong convexity and smoothness.

Accelerated convergence in the convex case.

Improved distributed optimization with compressed communication.

Abstract

Stochastic gradient descent type methods are ubiquitous in machine learning, but they are only applicable to the optimization of differentiable functions. Proximal algorithms are more general and applicable to nonsmooth functions. We propose a new stochastic and variance-reduced algorithm, the Stochastic Multi-Proximal Method (SMPM), in which the proximity operators of a (possibly empty) random subset of functions are called at every iteration, according to an arbitrary sampling distribution. Several existing algorithms, including Point-SAGA (2016), Proxskip (2022) and RandProx-Minibatch (2023) are recovered as particular cases. We derive linear convergence results in presence of strong convexity and smoothness or similarity of the functions. We prove convergence in the general convex case and accelerated O(1/t2) convergence with varying stepsizes in presence of strong convexity solely. Our results are new even for the above special cases. Moreover, we show an application to distributed optimization with compressed communication, outperforming existing methods.