Federated Progressive Self-Distillation with Logits Calibration for Personalized IIoT Edge Intelligence

TL;DR

This paper introduces FedPSD, a federated learning method for IIoT that uses logits calibration and progressive self-distillation to mitigate forgetting of personalized and global knowledge during local training.

Contribution

It proposes a novel federated learning approach that addresses knowledge forgetting through logits calibration and progressive self-distillation, enhancing personalization in IIoT applications.

Findings

FedPSD effectively reduces knowledge forgetting in heterogeneous data scenarios.

The method outperforms existing PFL techniques in personalization accuracy.

Extensive experiments validate the superiority of FedPSD in diverse data distributions.

Abstract

Personalized Federated Learning (PFL) focuses on tailoring models to individual IIoT clients in federated learning by addressing data heterogeneity and diverse user needs. Although existing studies have proposed effective PFL solutions from various perspectives, they overlook the issue of forgetting both historical personalized knowledge and global generalized knowledge during local training on clients. Therefore, this study proposes a novel PFL method, Federated Progressive Self-Distillation (FedPSD), based on logits calibration and progressive self-distillation. We analyze the impact mechanism of client data distribution characteristics on personalized and global knowledge forgetting. To address the issue of global knowledge forgetting, we propose a logits calibration approach for the local training loss and design a progressive self-distillation strategy to facilitate the gradual inheritance of global knowledge, where the model outputs from the previous epoch serve as virtual teachers to guide the training of subsequent epochs. Moreover, to address personalized knowledge forgetting, we construct calibrated fusion labels by integrating historical personalized model outputs, which are then used as teacher model outputs to guide the initial epoch of local self-distillation, enabling rapid recall of personalized knowledge. Extensive experiments under various data heterogeneity scenarios demonstrate the effectiveness and superiority of the proposed FedPSD method.