CYCle: Choosing Your Collaborators Wisely to Enhance Collaborative Fairness in Decentralized Learning

TL;DR

This paper introduces CYCle, a decentralized learning protocol that promotes fair collaboration gains among participants by using a novel reputation scoring method, improving fairness and performance especially in heterogeneous data settings.

Contribution

The paper proposes the CYCle protocol with a new reputation scoring method based on gradient alignment, extending decentralized learning fairness beyond existing correlation-based measures.

Findings

CYCle achieves fairer collaboration gains in decentralized learning.

CYCle outperforms FedAvg in high heterogeneity scenarios.

Empirical results show positive, fair gains even with skewed data distributions.

Abstract

Collaborative learning (CL) enables multiple participants to jointly train machine learning (ML) models on decentralized data sources without raw data sharing. While the primary goal of CL is to maximize the expected accuracy gain for each participant, it is also important to ensure that the gains are fairly distributed: no client should be negatively impacted, and gains should reflect contributions. Most existing CL methods require central coordination and focus only on gain maximization, overlooking fairness. In this work, we first show that the existing measure of collaborative fairness based on the correlation between accuracy values without and with collaboration has drawbacks because it does not account for negative collaboration gain. We argue that maximizing mean collaboration gain (MCG) while simultaneously minimizing the collaboration gain spread (CGS) is a fairer alternative. Next, we propose the CYCle protocol that enables individual participants in a private decentralized learning (PDL) framework to achieve this objective through a novel reputation scoring method based on gradient alignment between the local cross-entropy and distillation losses. We further extend the CYCle protocol to operate on top of gossip-based decentralized algorithms such as Gossip-SGD. We also theoretically show that CYCle performs better than standard FedAvg in a two-client mean estimation setting under high heterogeneity. Empirical experiments demonstrate the effectiveness of the CYCle protocol to ensure positive and fair collaboration gain for all participants, even in cases where the data distributions of participants are highly skewed.