Contribution Evaluation in Federated Learning: Examining Current Approaches

TL;DR

This paper reviews and benchmarks current methods for evaluating individual contributions in federated learning, introducing a new approach to improve fairness and efficiency in contribution assessment.

Contribution

It provides a comprehensive review of existing contribution evaluation methods and introduces a novel approach, supported by benchmarking on standard datasets.

Findings

Existing CE methods vary in fairness and efficiency

The new approach outperforms some state-of-the-art methods

Benchmark results highlight strengths and weaknesses of different approaches

Abstract

Federated Learning (FL) has seen increasing interest in cases where entities want to collaboratively train models while maintaining privacy and governance over their data. In FL, clients with private and potentially heterogeneous data and compute resources come together to train a common model without raw data ever leaving their locale. Instead, the participants contribute by sharing local model updates, which, naturally, differ in quality. Quantitatively evaluating the worth of these contributions is termed the Contribution Evaluation (CE) problem. We review current CE approaches from the underlying mathematical framework to efficiently calculate a fair value for each client. Furthermore, we benchmark some of the most promising state-of-the-art approaches, along with a new one we introduce, on MNIST and CIFAR-10, to showcase their differences. Designing a fair and efficient CE method, while a small part of the overall FL system design, is tantamount to the mainstream adoption of FL.