Federated Skewed Label Learning with Logits Fusion

TL;DR

This paper introduces FedBalance, a federated learning method that calibrates logits using a private weak learner to address label distribution skew, significantly improving model accuracy across clients.

Contribution

FedBalance is a novel approach that corrects local model bias in federated learning with skewed labels by ensemble logits calibration, enhancing global model performance.

Findings

Achieves 13% higher average accuracy than state-of-the-art methods.

Effectively balances optimization among local models with label skew.

Improves global model robustness in heterogeneous data settings.

Abstract

Federated learning (FL) aims to collaboratively train a shared model across multiple clients without transmitting their local data. Data heterogeneity is a critical challenge in realistic FL settings, as it causes significant performance deterioration due to discrepancies in optimization among local models. In this work, we focus on label distribution skew, a common scenario in data heterogeneity, where the data label categories are imbalanced on each client. To address this issue, we propose FedBalance, which corrects the optimization bias among local models by calibrating their logits. Specifically, we introduce an extra private weak learner on the client side, which forms an ensemble model with the local model. By fusing the logits of the two models, the private weak learner can capture the variance of different data, regardless of their category. Therefore, the optimization direction of local models can be improved by increasing the penalty for misclassifying minority classes and reducing the attention to majority classes, resulting in a better global model. Extensive experiments show that our method can gain 13\% higher average accuracy compared with state-of-the-art methods.