High-Probability Convergence Theory for Distributed Composite Optimization with Sub-Weibull Noises

TL;DR

This paper introduces a distributed optimization framework that handles sub-Weibull noise, extending convergence guarantees to more realistic, heavy-tailed noise scenarios in multi-agent networks.

Contribution

It develops a new distributed stochastic mirror descent algorithm with sub-Weibull noise, providing high-probability convergence guarantees without smoothness assumptions.

Findings

Applicable in heavy-tailed noise environments

Guarantees high-probability convergence rates

Unified framework for various noise conditions

Abstract

With the rapid development of distributed optimization (DO) theory, the distributed stochastic gradient methods (DSGMs) occupy an important position. Although the theory of different DSGMs has been widely established, the main-stream results of existing work are still derived under the condition of light-tailed stochastic gradient noises. Increasing examples from various fields, indicate that, the light-tailed noise model is overly idealized in many practical instances, failing to capture the complexity and variability of noises in real-world scenarios, such as the presence of outliers or extreme values from data science and statistical learning. To address this issue, we propose a new DO framework that incorporates stochastic gradients under sub-Weibull randomness. We study a distributed composite stochastic mirror descent scheme with sub-Weibull gradient noise (DCSMD-SW) for solving a convex distributed composite optimization (DCO) problem over the time-varying multi-agent network. By investigating sub-Weibull randomness in DCSMD for the first time, we show that the algorithm is applicable in some common heavier-tailed noise environments while also guaranteeing good convergence properties. We comprehensively study the convergence performance of DCSMD-SW. Satisfactory high-probability convergence rates are derived for DCSMD-SW without any smoothness requirement. The work also offers a unified analytical framework for several critical cases of both algorithms and noise environments.