Take Your Pick: Enabling Effective Personalized Federated Learning within Low-dimensional Feature Space

TL;DR

This paper introduces FedPick, a novel personalized federated learning framework that operates in a low-dimensional feature space to adaptively select task-relevant features for each client, enhancing performance and interpretability.

Contribution

FedPick is the first method to perform personalized federated learning by selecting features in a low-dimensional space, addressing high-dimensional parameter challenges.

Findings

FedPick effectively selects task-relevant features for clients.

Improves model performance in cross-domain federated learning.

Offers greater interpretability compared to parameter-space methods.

Abstract

Personalized federated learning (PFL) is a popular framework that allows clients to have different models to address application scenarios where clients' data are in different domains. The typical model of a client in PFL features a global encoder trained by all clients to extract universal features from the raw data and personalized layers (e.g., a classifier) trained using the client's local data. Nonetheless, due to the differences between the data distributions of different clients (aka, domain gaps), the universal features produced by the global encoder largely encompass numerous components irrelevant to a certain client's local task. Some recent PFL methods address the above problem by personalizing specific parameters within the encoder. However, these methods encounter substantial challenges attributed to the high dimensionality and non-linearity of neural network parameter space. In contrast, the feature space exhibits a lower dimensionality, providing greater intuitiveness and interpretability as compared to the parameter space. To this end, we propose a novel PFL framework named FedPick. FedPick achieves PFL in the low-dimensional feature space by selecting task-relevant features adaptively for each client from the features generated by the global encoder based on its local data distribution. It presents a more accessible and interpretable implementation of PFL compared to those methods working in the parameter space. Extensive experimental results show that FedPick could effectively select task-relevant features for each client and improve model performance in cross-domain FL.