Beyond performance-wise Contribution Evaluation in Federated Learning

TL;DR

This paper explores a comprehensive evaluation of client contributions in federated learning, emphasizing trustworthiness aspects like reliability, resilience, and fairness, beyond traditional performance metrics, using Shapley value approximations.

Contribution

It introduces a multifaceted contribution assessment framework in federated learning, considering trustworthiness dimensions and employing Shapley value for attribution, revealing the inadequacy of single-metric evaluations.

Findings

Clients vary significantly across trustworthiness dimensions.

Current metrics do not capture all aspects of contribution.

No single client excels in all evaluated dimensions.

Abstract

Federated learning offers a privacy-friendly collaborative learning framework, yet its success, like any joint venture, hinges on the contributions of its participants. Existing client evaluation methods predominantly focus on model performance, such as accuracy or loss, which represents only one dimension of a machine learning model's overall utility. In contrast, this work investigates the critical, yet overlooked, issue of client contributions towards a model's trustworthiness -- specifically, its reliability (tolerance to noisy data), resilience (resistance to adversarial examples), and fairness (measured via demographic parity). To quantify these multifaceted contributions, we employ the state-of-the-art approximation of the Shapley value, a principled method for value attribution. Our results reveal that no single client excels across all dimensions, which are largely independent from each other, highlighting a critical flaw in current evaluation scheme: no single metric is adequate for comprehensive evaluation and equitable rewarding allocation.