Analysis of Decentralized Stochastic Successive Convex Approximation for composite non-convex problems

TL;DR

This paper introduces an accelerated decentralized stochastic successive convex approximation method that achieves optimal first-order complexity for non-convex problems, matching centralized lower bounds.

Contribution

It proposes the D-MSSCA algorithm, an accelerated decentralized SCA method with proven optimal stochastic first-order complexity for non-convex optimization.

Findings

D-MSSCA attains an SFO complexity of O(ε^{-3/2})

The method matches centralized lower bounds for stochastic non-convex optimization

The algorithm effectively handles convex constraints and non-smooth regularizers.

Abstract

This work considers the decentralized successive convex approximation (SCA) method for minimizing stochastic non-convex objectives subject to convex constraints, along with possibly non-smooth convex regularizers. Although SCA has been widely applied in decentralized settings, its stochastic first order (SFO) complexity is unknown, and it is thought to be slower than the centralized momentum-enhanced SCA variants. In this work, we advance the state-of-the-art for SCA methods by proposing an accelerated variant, namely the \textbf{D}ecentralized \textbf{M}omentum-based \textbf{S}tochastic \textbf{SCA} (\textbf{D-MSSCA}) and analyze its SFO complexity. The proposed algorithm entails creating a stochastic surrogate of the objective at every iteration, which is minimized at each node separately. Remarkably, the D-MSSCA achieves an SFO complexity of $\mathcal{O}(\epsilon^{-3/2})$ to reach an $\epsilon$-stationary point, which is at par with the SFO complexity lower bound for unconstrained stochastic non-convex optimization in centralized setting.