Communication Efficient Federated Learning via Ordered ADMM in a Fully Decentralized Setting

TL;DR

This paper introduces OADMM, a communication-efficient decentralized algorithm for distributed optimization that orders message transmissions based on data informativeness, reducing communication costs and accelerating convergence.

Contribution

It proposes a novel ordered ADMM algorithm for fully decentralized networks, improving communication efficiency and convergence speed over classical ADMM.

Findings

OADMM significantly reduces communication compared to existing algorithms.

SOADMM accelerates convergence and further reduces communication.

Numerical results validate the effectiveness of the proposed methods.

Abstract

The challenge of communication-efficient distributed optimization has attracted attention in recent years. In this paper, a communication efficient algorithm, called ordering-based alternating direction method of multipliers (OADMM) is devised in a general fully decentralized network setting where a worker can only exchange messages with neighbors. Compared to the classical ADMM, a key feature of OADMM is that transmissions are ordered among workers at each iteration such that a worker with the most informative data broadcasts its local variable to neighbors first, and neighbors who have not transmitted yet can update their local variables based on that received transmission. In OADMM, we prohibit workers from transmitting if their current local variables are not sufficiently different from their previously transmitted value. A variant of OADMM, called SOADMM, is proposed where transmissions are ordered but transmissions are never stopped for each node at each iteration. Numerical results demonstrate that given a targeted accuracy, OADMM can significantly reduce the number of communications compared to existing algorithms including ADMM. We also show numerically that SOADMM can accelerate convergence, resulting in communication savings compared to the classical ADMM.