Jointly Computation- and Communication-Efficient Distributed Learning

TL;DR

This paper introduces a new ADMM-based distributed learning algorithm that enhances computation and communication efficiency through stochastic gradients, multiple local epochs, and compression, with proven linear convergence.

Contribution

It presents a novel joint computation- and communication-efficient ADMM algorithm with theoretical convergence guarantees for distributed learning.

Findings

Proven linear convergence in strongly convex settings.

Numerical results outperforming state-of-the-art methods.

Effective use of stochastic gradients and compression.

Abstract

We address distributed learning problems over undirected networks. Specifically, we focus on designing a novel ADMM-based algorithm that is jointly computation- and communication-efficient. Our design guarantees computational efficiency by allowing agents to use stochastic gradients during local training. Moreover, communication efficiency is achieved as follows: i) the agents perform multiple training epochs between communication rounds, and ii) compressed transmissions are used. We prove exact linear convergence of the algorithm in the strongly convex setting. We corroborate our theoretical results by numerical comparisons with state of the art techniques on a classification task.