Decentralized Non-convex Stochastic Optimization with Heterogeneous Variance

TL;DR

This paper introduces a decentralized stochastic optimization algorithm that adaptively handles heterogeneous variances across nodes, providing tighter complexity bounds and demonstrating optimality through theoretical analysis and experiments.

Contribution

It proposes D-NSS with node-specific sampling and D-NSS-VR with variance reduction, achieving optimal sample complexity bounds under heterogeneous variance conditions.

Findings

D-NSS achieves tighter bounds based on the mean of local variances.

D-NSS-VR improves sample complexity under mean-squared smoothness.

Numerical results confirm theoretical advantages and effectiveness.

Abstract

Decentralized optimization is critical for solving large-scale machine learning problems over distributed networks, where multiple nodes collaborate through local communication. In practice, the variances of stochastic gradient estimators often differ across nodes, yet their impact on algorithm design and complexity remains unclear. To address this issue, we propose D-NSS, a decentralized algorithm with node-specific sampling, and establish its sample complexity depending on the arithmetic mean of local standard deviations, achieving tighter bounds than existing methods that rely on the worst-case or quadratic mean. We further derive a matching sample complexity lower bound under heterogeneous variance, thereby proving the optimality of this dependence. Moreover, we extend the framework with a variance reduction technique and develop D-NSS-VR, which under the mean-squared smoothness assumption attains an improved sample complexity bound while preserving the arithmetic-mean dependence. Finally, numerical experiments validate the theoretical results and demonstrate the effectiveness of the proposed algorithms.