Personalized Federated Learning via Sequential Layer Expansion in Representation Learning

TL;DR

This paper introduces a novel personalized federated learning method that uses sequential layer expansion and layer scheduling to better handle data and class heterogeneity, improving accuracy and reducing costs.

Contribution

It proposes a new layer decoupling and scheduling approach in federated learning that enhances personalization and efficiency over existing methods.

Findings

Improved accuracy under heterogeneous data conditions

Reduced computation costs compared to existing algorithms

Effective handling of class heterogeneity through layer scheduling

Abstract

Federated learning ensures the privacy of clients by conducting distributed training on individual client devices and sharing only the model weights with a central server. However, in real-world scenarios, the heterogeneity of data among clients necessitates appropriate personalization methods. In this paper, we aim to address this heterogeneity using a form of parameter decoupling known as representation learning. Representation learning divides deep learning models into 'base' and 'head' components. The base component, capturing common features across all clients, is shared with the server, while the head component, capturing unique features specific to individual clients, remains local. We propose a new representation learning-based approach that suggests decoupling the entire deep learning model into more densely divided parts with the application of suitable scheduling methods, which can benefit not only data heterogeneity but also class heterogeneity. In this paper, we compare and analyze two layer scheduling approaches, namely forward (\textit{Vanilla}) and backward (\textit{Anti}), in the context of data and class heterogeneity among clients. Our experimental results show that the proposed algorithm, when compared to existing personalized federated learning algorithms, achieves increased accuracy, especially under challenging conditions, while reducing computation costs.