ROSS: RObust decentralized Stochastic learning based on Shapley values

TL;DR

ROSS is a robust decentralized learning algorithm that uses Shapley values to weight contributions, effectively handling heterogeneous, noisy, and poisoned data across agents, and demonstrating superior convergence and accuracy.

Contribution

The paper introduces ROSS, a novel decentralized stochastic learning method leveraging Shapley values for robustness against data heterogeneity and poisoning.

Findings

Achieves linear convergence speedup.

Outperforms state-of-the-art methods in convergence.

Improves prediction accuracy under challenging data conditions.

Abstract

In the paradigm of decentralized learning, a group of agents collaborate to learn a global model using a distributed dataset without a central server; nevertheless, it is severely challenged by the heterogeneity of the data distribution across the agents. For example, the data may be distributed non-independently and identically, and even be noised or poisoned. To address these data challenges, we propose ROSS, a novel robust decentralized stochastic learning algorithm based on Shapley values, in this paper. Specifically, in each round, each agent aggregates the cross-gradient information from its neighbors, i.e., the derivatives of its local model with respect to the datasets of its neighbors, to update its local model in a momentum like manner, while we innovate in weighting the derivatives according to their contributions measured by Shapley values. We perform solid theoretical analysis to reveal the linear convergence speedup of our ROSS algorithm. We also verify the efficacy of our algorithm through extensive experiments on public datasets. Our results demonstrate that, in face of the above variety of data challenges, our ROSS algorithm has significant advantages over existing state-of-the-art proposals in terms of both convergence and prediction accuracy.