Communication-Efficient Design for Quantized Decentralized Federated Learning

TL;DR

This paper introduces a novel non-uniform quantization method for decentralized federated learning, significantly reducing communication rounds and improving efficiency without assuming convex loss functions.

Contribution

It proposes the Lloyd-Max based quantization (LM-DFL) and a doubly-adaptive DFL approach, enhancing convergence and communication efficiency in decentralized federated learning.

Findings

LM-DFL minimizes quantization distortion effectively.

Doubly-adaptive DFL reduces communication by adjusting quantization levels.

Experimental results show improved efficiency on MNIST and CIFAR-10.

Abstract

Decentralized federated learning (DFL) is a variant of federated learning, where edge nodes only communicate with their one-hop neighbors to learn the optimal model. However, as information exchange is restricted in a range of one-hop in DFL, inefficient information exchange leads to more communication rounds to reach the targeted training loss. This greatly reduces the communication efficiency. In this paper, we propose a new non-uniform quantization of model parameters to improve DFL convergence. Specifically, we apply the Lloyd-Max algorithm to DFL (LM-DFL) first to minimize the quantization distortion by adjusting the quantization levels adaptively. Convergence guarantee of LM-DFL is established without convex loss assumption. Based on LM-DFL, we then propose a new doubly-adaptive DFL, which jointly considers the ascending number of quantization levels to reduce the amount of communicated information in the training and adapts the quantization levels for non-uniform gradient distributions. Experiment results based on MNIST and CIFAR-10 datasets illustrate the superiority of LM-DFL with the optimal quantized distortion and show that doubly-adaptive DFL can greatly improve communication efficiency.