FedHQ: Hybrid Runtime Quantization for Federated Learning

TL;DR

FedHQ introduces a hybrid quantization framework combining PTQ and QAT to enhance federated learning efficiency and accuracy, addressing device and data heterogeneity with adaptive strategy allocation.

Contribution

This paper presents FedHQ, a novel framework that automatically optimizes hybrid quantization strategies for federated learning, balancing speed and accuracy across diverse settings.

Findings

Achieves up to 2.47x training acceleration.

Improves accuracy by up to 11.15%.

Maintains negligible additional overhead.

Abstract

Federated Learning (FL) is a decentralized model training approach that preserves data privacy but struggles with low efficiency. Quantization, a powerful training optimization technique, has been widely explored for integration into FL. However, many studies fail to consider the distinct performance attribution between particular quantization strategies, such as post-training quantization (PTQ) or quantization-aware training (QAT). As a result, existing FL quantization methods rely solely on either PTQ or QAT, optimizing for speed or accuracy while compromising the other. To efficiently accelerate FL and maintain distributed convergence accuracy across various FL settings, this paper proposes a hybrid quantitation approach combining PTQ and QAT for FL systems. We conduct case studies to validate the effectiveness of using hybrid quantization in FL. To solve the difficulty of modeling speed and accuracy caused by device and data heterogeneity, we propose a hardware-related analysis and data-distribution-related analysis to help identify the trade-off boundaries for strategy selection. Based on these, we proposed a novel framework named FedHQ to automatically adopt optimal hybrid strategy allocation for FL systems. Specifically, FedHQ develops a coarse-grained global initialization and fine-grained ML-based adjustment to ensure efficiency and robustness. Experiments show that FedHQ achieves up to 2.47x times training acceleration and up to 11.15% accuracy improvement and negligible extra overhead.