A Stochastic Proximal Gradient Framework for Decentralized Non-Convex Composite Optimization: Topology-Independent Sample Complexity and Communication Efficiency

TL;DR

This paper introduces ProxGT, a stochastic proximal gradient framework for decentralized non-convex composite optimization, achieving topology-independent sample complexity and linear speedups over centralized methods.

Contribution

The paper presents the first provably efficient decentralized stochastic proximal gradient method with topology-independent sample complexity and improved communication efficiency.

Findings

Sample complexities are network topology-independent.

Achieves linear speedups compared to centralized methods.

Framework applicable to various non-convex composite problems.

Abstract

Decentralized optimization is a promising parallel computation paradigm for large-scale data analytics and machine learning problems defined over a network of nodes. This paper is concerned with decentralized non-convex composite problems with population or empirical risk. In particular, the networked nodes are tasked to find an approximate stationary point of the average of local, smooth, possibly non-convex risk functions plus a possibly non-differentiable extended valued convex regularizer. Under this general formulation, we propose the first provably efficient, stochastic proximal gradient framework, called ProxGT. Specifically, we construct and analyze several instances of ProxGT that are tailored respectively for different problem classes of interest. Remarkably, we show that the sample complexities of these instances are network topology-independent and achieve linear speedups compared to that of the corresponding centralized optimal methods implemented on a single node.