Communication-Efficient Distributed Asynchronous ADMM

TL;DR

This paper proposes a communication-efficient asynchronous ADMM method using coarse quantization to reduce data exchange costs in large-scale distributed learning and optimization, with verified convergence on neural network tasks.

Contribution

Introducing coarse quantization into asynchronous ADMM to significantly reduce communication costs in federated learning and distributed optimization.

Findings

Convergence verified for multiple distributed learning tasks.

Effective reduction in communication overhead demonstrated.

Applicable to neural network training scenarios.

Abstract

In distributed optimization and federated learning, asynchronous alternating direction method of multipliers (ADMM) serves as an attractive option for large-scale optimization, data privacy, straggler nodes and variety of objective functions. However, communication costs can become a major bottleneck when the nodes have limited communication budgets or when the data to be communicated is prohibitively large. In this work, we propose introducing coarse quantization to the data to be exchanged in aynchronous ADMM so as to reduce communication overhead for large-scale federated learning and distributed optimization applications. We experimentally verify the convergence of the proposed method for several distributed learning tasks, including neural networks.