Distributed Stochastic Consensus Optimization with Momentum for Nonconvex Nonsmooth Problems

TL;DR

This paper introduces a novel distributed stochastic optimization algorithm with momentum for non-convex, non-smooth problems, achieving optimal communication complexity and demonstrating effectiveness in neural network training.

Contribution

It presents the first stochastic distributed algorithm with $ ext{O}(1/ ext{epsilon})$ communication complexity for non-convex, non-smooth problems, using a proximal primal-dual approach with Nesterov momentum.

Findings

Achieves $ ext{O}(1/ ext{epsilon}^2)$ computation complexity for $ ext{epsilon}$-stationary solutions.

Attains $ ext{O}(1/ ext{epsilon})$ communication complexity, lower than existing methods.

Effectively applied to distributed non-convex regression and neural network classification.

Abstract

While many distributed optimization algorithms have been proposed for solving smooth or convex problems over the networks, few of them can handle non-convex and non-smooth problems. Based on a proximal primal-dual approach, this paper presents a new (stochastic) distributed algorithm with Nesterov momentum for accelerated optimization of non-convex and non-smooth problems. Theoretically, we show that the proposed algorithm can achieve an $\epsilon$-stationary solution under a constant step size with $\mathcal{O}(1/\epsilon^2)$ computation complexity and $\mathcal{O}(1/\epsilon)$ communication complexity. When compared to the existing gradient tracking based methods, the proposed algorithm has the same order of computation complexity but lower order of communication complexity. To the best of our knowledge, the presented result is the first stochastic algorithm with the $\mathcal{O}(1/\epsilon)$ communication complexity for non-convex and non-smooth problems. Numerical experiments for a distributed non-convex regression problem and a deep neural network based classification problem are presented to illustrate the effectiveness of the proposed algorithms.