On the Volatility of Shapley-Based Contribution Metrics in Federated Learning

TL;DR

This paper investigates the stability of Shapley-value-based contribution metrics in federated learning, revealing significant fluctuations across different aggregation strategies and data distributions, which impacts fair participant reward allocation.

Contribution

The study provides a comprehensive analysis of the volatility of Shapley values in federated learning, highlighting their instability across various aggregation methods and data heterogeneity scenarios.

Findings

Shapley values show high variability across aggregation strategies.

Contribution metrics are unstable in non-IID data settings.

Gradient-based reconstruction methods reveal discrepancies in participant contributions.

Abstract

Federated learning (FL) is a collaborative and privacy-preserving Machine Learning paradigm, allowing the development of robust models without the need to centralize sensitive data. A critical challenge in FL lies in fairly and accurately allocating contributions from diverse participants. Inaccurate allocation can undermine trust, lead to unfair compensation, and thus participants may lack the incentive to join or actively contribute to the federation. Various remuneration strategies have been proposed to date, including auction-based approaches and Shapley-value-based methods, the latter offering a means to quantify the contribution of each participant. However, little to no work has studied the stability of these contribution evaluation methods. In this paper, we evaluate participant contributions in federated learning using gradient-based model reconstruction techniques with Shapley values and compare the round-based contributions to a classic data contribution measurement scheme. We provide an extensive analysis of the discrepancies of Shapley values across a set of aggregation strategies and examine them on an overall and a per-client level. We show that, between different aggregation techniques, Shapley values lead to unstable reward allocations among participants. Our analysis spans various data heterogeneity distributions, including independent and identically distributed (IID) and non-IID scenarios.