Stochastic Decentralized Optimization of Non-Smooth Convex and Convex-Concave Problems over Time-Varying Networks

TL;DR

This paper develops decentralized algorithms for non-smooth stochastic convex and saddle point problems over dynamic networks, providing bounds on computation and communication that match theoretical lower limits.

Contribution

It extends existing decentralized optimization theory to non-smooth, stochastic, and time-varying network scenarios, including saddle point problems.

Findings

Established upper bounds on oracle calls and communication rounds.

Extended theory to non-smooth stochastic and saddle point problems.

Matched bounds with known lower limits for these problems.

Abstract

We study non-smooth stochastic decentralized optimization problems over time-varying networks, where objective functions are distributed across nodes and network connections may intermittently appear or break. Specifically, we consider two settings: (i) stochastic non-smooth (strongly) convex optimization, and (ii) stochastic non-smooth (strongly) convex-(strongly) concave saddle point optimization. Convex problems of this type commonly arise in deep neural network training, while saddle point problems are central to machine learning tasks such as the training of generative adversarial networks (GANs). Prior works have primarily focused on the smooth setting, or time-invariant network scenarios. We extend the existing theory to the more general non-smooth and stochastic setting over time-varying networks and saddle point problems. Our analysis establishes upper bounds on both the number of stochastic oracle calls and communication rounds, matching lower bounds for both convex and saddle point optimization problems.