Federated Learning via Consensus Mechanism on Heterogeneous Data: A New Perspective on Convergence

TL;DR

This paper introduces FedCOME, a federated learning method that uses a consensus mechanism to ensure risk reduction for individual clients on heterogeneous data, with proven convergence and improved empirical performance.

Contribution

The paper proposes FedCOME, a novel consensus-based approach that guarantees client risk decrease and convergence in heterogeneous federated learning scenarios.

Findings

FedCOME guarantees convergence of the global objective.

Empirically outperforms state-of-the-art methods on benchmark datasets.

Enhances fairness and efficiency in federated learning.

Abstract

Federated learning (FL) on heterogeneous data (non-IID data) has recently received great attention. Most existing methods focus on studying the convergence guarantees for the global objective. While these methods can guarantee the decrease of the global objective in each communication round, they fail to ensure risk decrease for each client. In this paper, to address the problem,we propose FedCOME, which introduces a consensus mechanism to enforce decreased risk for each client after each training round. In particular, we allow a slight adjustment to a client's gradient on the server side, which generates an acute angle between the corrected gradient and the original ones of other clients. We theoretically show that the consensus mechanism can guarantee the convergence of the global objective. To generalize the consensus mechanism to the partial participation FL scenario, we devise a novel client sampling strategy to select the most representative clients for the global data distribution. Training on these selected clients with the consensus mechanism could empirically lead to risk decrease for clients that are not selected. Finally, we conduct extensive experiments on four benchmark datasets to show the superiority of FedCOME against other state-of-the-art methods in terms of effectiveness, efficiency and fairness. For reproducibility, we make our source code publicly available at: \url{https://github.com/fedcome/fedcome}.