Timely Parameter Updating in Over-the-Air Federated Learning

TL;DR

This paper introduces FAIR-k, a novel algorithm for over-the-air federated learning that intelligently selects impactful gradient updates to improve convergence speed and communication efficiency amidst high-dimensional models and channel noise.

Contribution

FAIR-k combines round-robin and top-k strategies to select impactful gradients, addressing the limited orthogonal waveforms issue in over-the-air federated learning and providing convergence analysis under data heterogeneity.

Findings

FAIR-k accelerates convergence compared to existing methods.

It improves communication efficiency by reducing the number of transmitted gradients.

The analysis accounts for data heterogeneity, channel noise, and parameter staleness.

Abstract

Incorporating over-the-air computations (OAC) into the model training process of federated learning (FL) is an effective approach to alleviating the communication bottleneck in FL systems. Under OAC-FL, every client modulates its intermediate parameters, such as gradient, onto the same set of orthogonal waveforms and simultaneously transmits the radio signal to the edge server. By exploiting the superposition property of multiple-access channels, the edge server can obtain an automatically aggregated global gradient from the received signal. However, the limited number of orthogonal waveforms available in practical systems is fundamentally mismatched with the high dimensionality of modern deep learning models. To address this issue, we propose Freshness Freshness-mAgnItude awaRe top-k (FAIR-k), an algorithm that selects, in each communication round, the most impactful subset of gradients to be updated over the air. In essence, FAIR-k combines the complementary strengths of the Round-Robin and Top-k algorithms, striking a delicate balance between timeliness (freshness of parameter updates) and importance (gradient magnitude). Leveraging tools from Markov analysis, we characterize the distribution of parameter staleness under FAIR-k. Building on this, we establish the convergence rate of OAC-FL with FAIR-k, which discloses the joint effect of data heterogeneity, channel noise, and parameter staleness on the training efficiency. Notably, as opposed to conventional analyses that assume a universal Lipschitz constant across all the clients, our framework adopts a finer-grained model of the data heterogeneity. The analysis demonstrates that since FAIR-k promotes fresh (and fair) parameter updates, it not only accelerates convergence but also enhances communication efficiency by enabling an extended period of local training without significantly affecting overall training efficiency.