Partial Knowledge Distillation for Alleviating the Inherent Inter-Class Discrepancy in Federated Learning

TL;DR

This paper investigates the persistent accuracy gap among classes in federated learning, identifies inherent weak classes, and proposes a partial knowledge distillation method to improve their classification performance.

Contribution

It introduces the concept of inherent weak classes in federated learning and proposes PKD to enhance their accuracy, addressing a previously overlooked issue.

Findings

Inherent weak classes exist even in balanced datasets.

PKD improves weak class accuracy by 10.7%.

Reduces inter-class accuracy discrepancy effectively.

Abstract

Substantial efforts have been devoted to alleviating the impact of the long-tailed class distribution in federated learning. In this work, we observe an interesting phenomenon that certain weak classes consistently exist even for class-balanced learning. These weak classes, different from the minority classes in the previous works, are inherent to data and remain fairly consistent for various network structures, learning paradigms, and data partitioning methods. The inherent inter-class accuracy discrepancy can reach over 36.9% for federated learning on the FashionMNIST and CIFAR-10 datasets, even when the class distribution is balanced both globally and locally. In this study, we empirically analyze the potential reason for this phenomenon. Furthermore, a partial knowledge distillation (PKD) method is proposed to improve the model's classification accuracy for weak classes. In this approach, knowledge transfer is initiated upon the occurrence of specific misclassifications within certain weak classes. Experimental results show that the accuracy of weak classes can be improved by 10.7%, reducing the inherent inter-class discrepancy effectively.