Federated Learning under Partially Class-Disjoint Data via Manifold Reshaping

TL;DR

This paper introduces FedMR, a manifold reshaping method that improves federated learning with partially class-disjoint data by calibrating feature space to prevent collapse and enhance class separation.

Contribution

It proposes a novel manifold reshaping approach with intra- and inter-class losses to address class bias in federated learning with PCDD, improving accuracy and efficiency.

Findings

FedMR significantly outperforms baseline methods in accuracy.

It enhances communication efficiency in federated learning.

The approach effectively mitigates class bias in PCDD scenarios.

Abstract

Statistical heterogeneity severely limits the performance of federated learning (FL), motivating several explorations e.g., FedProx, MOON and FedDyn, to alleviate this problem. Despite effectiveness, their considered scenario generally requires samples from almost all classes during the local training of each client, although some covariate shifts may exist among clients. In fact, the natural case of partially class-disjoint data (PCDD), where each client contributes a few classes (instead of all classes) of samples, is practical yet underexplored. Specifically, the unique collapse and invasion characteristics of PCDD can induce the biased optimization direction in local training, which prevents the efficiency of federated learning. To address this dilemma, we propose a manifold reshaping approach called FedMR to calibrate the feature space of local training. Our FedMR adds two interplaying losses to the vanilla federated learning: one is intra-class loss to decorrelate feature dimensions for anti-collapse; and the other one is inter-class loss to guarantee the proper margin among categories in the feature expansion. We conduct extensive experiments on a range of datasets to demonstrate that our FedMR achieves much higher accuracy and better communication efficiency. Source code is available at: https://github.com/MediaBrain-SJTU/FedMR.git.