Bitwidth Heterogeneous Federated Learning with Progressive Weight Dequantization

TL;DR

This paper introduces a new federated learning scenario with devices having different bitwidths, and proposes a progressive weight dequantization method to effectively aggregate and reconstruct model weights, improving performance.

Contribution

The paper presents BHFL, a novel federated learning setting with bitwidth heterogeneity, and proposes ProWD, a framework with a trainable dequantizer for better weight aggregation.

Findings

ProWD outperforms baseline federated learning algorithms.

ProWD effectively reconstructs high-precision weights from low-bitwidth models.

The approach mitigates performance degradation caused by bitwidth heterogeneity.

Abstract

In practical federated learning scenarios, the participating devices may have different bitwidths for computation and memory storage by design. However, despite the progress made in device-heterogeneous federated learning scenarios, the heterogeneity in the bitwidth specifications in the hardware has been mostly overlooked. We introduce a pragmatic FL scenario with bitwidth heterogeneity across the participating devices, dubbed as Bitwidth Heterogeneous Federated Learning (BHFL). BHFL brings in a new challenge, that the aggregation of model parameters with different bitwidths could result in severe performance degeneration, especially for high-bitwidth models. To tackle this problem, we propose ProWD framework, which has a trainable weight dequantizer at the central server that progressively reconstructs the low-bitwidth weights into higher bitwidth weights, and finally into full-precision weights. ProWD further selectively aggregates the model parameters to maximize the compatibility across bit-heterogeneous weights. We validate ProWD against relevant FL baselines on the benchmark datasets, using clients with varying bitwidths. Our ProWD largely outperforms the baseline FL algorithms as well as naive approaches (e.g. grouped averaging) under the proposed BHFL scenario.