Distributed gradient methods under heavy-tailed communication noise

TL;DR

This paper introduces a novel distributed gradient method designed to operate effectively under heavy-tailed communication noise, common in wireless sensor and IoT networks, ensuring convergence and robustness where traditional methods fail.

Contribution

It proposes a new distributed optimization algorithm with a mixed time-scale approach and nonlinear consensus updates that handle heavy-tailed noise, a first in this context.

Findings

Converges to a neighborhood of the optimal solution in mean squared error sense.

The asymptotic MSE can be minimized through step-size tuning.

Outperforms existing methods under heavy-tailed noise conditions.

Abstract

We consider a standard distributed optimization problem in which networked nodes collaboratively minimize the sum of their locally known convex costs. For this setting, we address for the first time the fundamental problem of design and analysis of distributed methods to solve the above problem when inter-node communication is subject to \emph{heavy-tailed} noise. Heavy-tailed noise is highly relevant and frequently arises in densely deployed wireless sensor and Internet of Things (IoT) networks. Specifically, we design a distributed gradient-type method that features a carefully balanced mixed time-scale time-varying consensus and gradient contribution step sizes and a bounded nonlinear operator on the consensus update to limit the effect of heavy-tailed noise. Assuming heterogeneous strongly convex local costs with mutually different minimizers that are arbitrarily far apart, we show that the proposed method converges to a neighborhood of the network-wide problem solution in the mean squared error (MSE) sense, and we also characterize the corresponding convergence rate. We further show that the asymptotic MSE can be made arbitrarily small through consensus step-size tuning, possibly at the cost of slowing down the transient error decay. Numerical experiments corroborate our findings and demonstrate the resilience of the proposed method to heavy-tailed (and infinite variance) communication noise. They also show that existing distributed methods, designed for finite-communication-noise-variance settings, fail in the presence of infinite variance noise.