CyclicFL: A Cyclic Model Pre-Training Approach to Efficient Federated Learning

TL;DR

CyclicFL is a novel federated learning pre-training approach that enhances convergence speed and accuracy by deriving effective initial models without exposing local data, especially benefiting non-IID scenarios.

Contribution

The paper introduces CyclicFL, a cyclic pre-training method for FL that improves convergence and accuracy without requiring public data, and provides theoretical analysis of its effectiveness.

Findings

Up to 14.11% increase in classification accuracy.

Significantly faster convergence in FL training.

Effective on non-IID data scenarios.

Abstract

Federated learning (FL) has been proposed to enable distributed learning on Artificial Intelligence Internet of Things (AIoT) devices with guarantees of high-level data privacy. Since random initial models in FL can easily result in unregulated Stochastic Gradient Descent (SGD) processes, existing FL methods greatly suffer from both slow convergence and poor accuracy, especially in non-IID scenarios. To address this problem, we propose a novel method named CyclicFL, which can quickly derive effective initial models to guide the SGD processes, thus improving the overall FL training performance. We formally analyze the significance of data consistency between the pre-training and training stages of CyclicFL, showing the limited Lipschitzness of loss for the pre-trained models by CyclicFL. Moreover, we systematically prove that our method can achieve faster convergence speed under various convexity assumptions. Unlike traditional centralized pre-training methods that require public proxy data, CyclicFL pre-trains initial models on selected AIoT devices cyclically without exposing their local data. Therefore, they can be easily integrated into any security-critical FL methods. Comprehensive experimental results show that CyclicFL can not only improve the maximum classification accuracy by up to $14.11\%$ but also significantly accelerate the overall FL training process.