Fast and Efficient Gossip Algorithms for Robust and Non-smooth Decentralized Learning

TL;DR

This paper introduces AsylADMM, an asynchronous gossip algorithm for decentralized non-smooth optimization that is memory-efficient and converges faster than existing methods, enabling robust learning on resource-limited edge devices.

Contribution

It presents a novel asynchronous gossip algorithm requiring only two variables per node for non-smooth optimization, with theoretical convergence analysis and empirical validation.

Findings

AsylADMM converges faster than existing baselines on non-smooth problems.

The algorithm requires only two variables per node, making it memory-efficient.

Empirical results include successful applications to quantile estimation, median estimation, and robust regression.

Abstract

Decentralized learning on resource-constrained edge devices demands algorithms that are communication-efficient, robust to data corruption, and lightweight in memory. State-of-the-art gossip-based methods address communication efficiency, but achieving robustness remains challenging. Methods for robust estimation and optimization typically rely on non-smooth objectives (\textit{e.g.}, pinball loss, $\ell_1$ loss), yet standard gossip methods are primarily designed for smooth losses. Asynchronous decentralized ADMM-based methods have been proposed to handle such non-smooth objectives; however, existing approaches require memory that scales with node degree, making them impractical when memory is limited. We propose AsylADMM, a novel asynchronous gossip algorithm for decentralized non-smooth optimization requiring only two variables per node. We provide a new theoretical analysis for the synchronous variant and leverage it to prove convergence of AsylADMM in a simplified setting based on the squared loss. Empirically, AsylADMM converges faster than existing baselines on challenging non-smooth problems, including quantile and geometric median estimation, lasso regression, and robust regression. More broadly, our novel gossip framework opens a practical pathway toward robust and non-smooth decentralized learning.