Gradual Federated Learning with Simulated Annealing

TL;DR

This paper introduces a novel federated learning method called SAFL that uses simulated annealing to improve convergence and accuracy, especially in early training stages with non-i.i.d. data distributions.

Contribution

The paper proposes a new federated learning approach utilizing simulated annealing to enhance convergence and performance over traditional FedAvg.

Findings

SAFL outperforms FedAvg in convergence speed.

SAFL achieves higher classification accuracy.

SAFL is effective with non-i.i.d. data distributions.

Abstract

Federated averaging (FedAvg) is a popular federated learning (FL) technique that updates the global model by averaging local models and then transmits the updated global model to devices for their local model update. One main limitation of FedAvg is that the average-based global model is not necessarily better than local models in the early stage of the training process so that FedAvg might diverge in realistic scenarios, especially when the data is non-identically distributed across devices and the number of data samples varies significantly from device to device. In this paper, we propose a new FL technique based on simulated annealing. The key idea of the proposed technique, henceforth referred to as \textit{simulated annealing-based FL} (SAFL), is to allow a device to choose its local model when the global model is immature. Specifically, by exploiting the simulated annealing strategy, we make each device choose its local model with high probability in early iterations when the global model is immature. From extensive numerical experiments using various benchmark datasets, we demonstrate that SAFL outperforms the conventional FedAvg technique in terms of the convergence speed and the classification accuracy.