Federated Learning-Distillation Alternation for Resource-Constrained IoT

TL;DR

This paper proposes FLDA, an alternating federated learning and distillation method for resource-constrained IoT devices, improving accuracy, convergence speed, and energy efficiency under interference and energy harvesting conditions.

Contribution

FLDA introduces an alternating approach between federated distillation and federated learning to optimize resource use and accuracy in IoT networks.

Findings

FLDA outperforms FL and FD in accuracy and convergence speed.

FLDA reduces energy consumption by up to 98%.

FLDA is more robust to interference than FL.

Abstract

Federated learning (FL) faces significant challenges in Internet of Things (IoT) networks due to device limitations in energy and communication resources, especially when considering the large size of FL models. From an energy perspective, the challenge is aggravated if devices rely on energy harvesting (EH), as energy availability can vary significantly over time, influencing the average number of participating users in each iteration. Additionally, the transmission of large model updates is more susceptible to interference from uncorrelated background traffic in shared wireless environments. As an alternative, federated distillation (FD) reduces communication overhead and energy consumption by transmitting local model outputs, which are typically much smaller than the entire model used in FL. However, this comes at the cost of reduced model accuracy. Therefore, in this paper, we propose FL-distillation alternation (FLDA). In FLDA, devices alternate between FD and FL phases, balancing model information with lower communication overhead and energy consumption per iteration. We consider a multichannel slotted-ALOHA EH-IoT network subject to background traffic/interference. In such a scenario, FLDA demonstrates higher model accuracy than both FL and FD, and achieves faster convergence than FL. Moreover, FLDA achieves target accuracies saving up to 98% in energy consumption, while also being less sensitive to interference, both relative to FL.