FedDisco: Federated Learning with Discrepancy-Aware Collaboration

TL;DR

FedDisco introduces a discrepancy-aware aggregation method for federated learning that improves performance by considering category distribution differences, outperforming existing methods and enhancing privacy, efficiency, and modularity.

Contribution

The paper proposes FedDisco, a novel federated learning aggregation method that incorporates discrepancy measures between local and global category distributions, backed by theoretical analysis and extensive experiments.

Findings

FedDisco outperforms several state-of-the-art methods.

Incorporating discrepancy improves aggregation effectiveness.

FedDisco enhances privacy, communication, and computation efficiency.

Abstract

This work considers the category distribution heterogeneity in federated learning. This issue is due to biased labeling preferences at multiple clients and is a typical setting of data heterogeneity. To alleviate this issue, most previous works consider either regularizing local models or fine-tuning the global model, while they ignore the adjustment of aggregation weights and simply assign weights based on the dataset size. However, based on our empirical observations and theoretical analysis, we find that the dataset size is not optimal and the discrepancy between local and global category distributions could be a beneficial and complementary indicator for determining aggregation weights. We thus propose a novel aggregation method, Federated Learning with Discrepancy-aware Collaboration (FedDisco), whose aggregation weights not only involve both the dataset size and the discrepancy value, but also contribute to a tighter theoretical upper bound of the optimization error. FedDisco also promotes privacy-preservation, communication and computation efficiency, as well as modularity. Extensive experiments show that our FedDisco outperforms several state-of-the-art methods and can be easily incorporated with many existing methods to further enhance the performance. Our code will be available at https://github.com/MediaBrain-SJTU/FedDisco.