Federated Learning with Label-Masking Distillation

TL;DR

This paper introduces FedLMD, a novel federated learning method that improves performance under label distribution skew by masking majority label predictions during distillation, effectively preserving minority label knowledge.

Contribution

The paper proposes FedLMD, a label-masking distillation technique for federated learning with skewed label distributions, including a resource-efficient variant FedLMD-Tf that outperforms existing methods.

Findings

FedLMD achieves state-of-the-art results across various scenarios.

The resource-efficient FedLMD-Tf outperforms previous lightweight approaches.

Masking majority labels enhances minority label knowledge preservation.

Abstract

Federated learning provides a privacy-preserving manner to collaboratively train models on data distributed over multiple local clients via the coordination of a global server. In this paper, we focus on label distribution skew in federated learning, where due to the different user behavior of the client, label distributions between different clients are significantly different. When faced with such cases, most existing methods will lead to a suboptimal optimization due to the inadequate utilization of label distribution information in clients. Inspired by this, we propose a label-masking distillation approach termed FedLMD to facilitate federated learning via perceiving the various label distributions of each client. We classify the labels into majority and minority labels based on the number of examples per class during training. The client model learns the knowledge of majority labels from local data. The process of distillation masks out the predictions of majority labels from the global model, so that it can focus more on preserving the minority label knowledge of the client. A series of experiments show that the proposed approach can achieve state-of-the-art performance in various cases. Moreover, considering the limited resources of the clients, we propose a variant FedLMD-Tf that does not require an additional teacher, which outperforms previous lightweight approaches without increasing computational costs. Our code is available at https://github.com/wnma3mz/FedLMD.