Spectral Co-Distillation for Personalized Federated Learning

TL;DR

This paper introduces spectral co-distillation, a novel method leveraging model spectrum information to improve personalized federated learning, along with a wait-free training protocol, demonstrating superior performance across diverse datasets.

Contribution

The paper proposes spectral co-distillation, a new spectral-based distillation approach, and a wait-free local training protocol for personalized federated learning, addressing data heterogeneity more effectively.

Findings

Spectral co-distillation outperforms existing PFL methods.

The wait-free training protocol reduces idle time during training.

Experimental results show improved personalization accuracy.

Abstract

Personalized federated learning (PFL) has been widely investigated to address the challenge of data heterogeneity, especially when a single generic model is inadequate in satisfying the diverse performance requirements of local clients simultaneously. Existing PFL methods are inherently based on the idea that the relations between the generic global and personalized local models are captured by the similarity of model weights. Such a similarity is primarily based on either partitioning the model architecture into generic versus personalized components, or modeling client relationships via model weights. To better capture similar (yet distinct) generic versus personalized model representations, we propose \textit{spectral distillation}, a novel distillation method based on model spectrum information. Building upon spectral distillation, we also introduce a co-distillation framework that establishes a two-way bridge between generic and personalized model training. Moreover, to utilize the local idle time in conventional PFL, we propose a wait-free local training protocol. Through extensive experiments on multiple datasets over diverse heterogeneous data settings, we demonstrate the outperformance and efficacy of our proposed spectral co-distillation method, as well as our wait-free training protocol.