FedRandom: Sampling Consistent and Accurate Contribution Values in Federated Learning

TL;DR

FedRandom is a novel method that improves the stability and accuracy of contribution evaluation in federated learning by generating additional samples, addressing the inherent instability issues of existing strategies.

Contribution

FedRandom introduces a statistical estimation approach that produces more samples for consistent contribution assessment in federated learning.

Findings

Reduces the distance to ground truth by over 33% in many scenarios

Improves contribution stability in over 90% of cases

Effective across multiple datasets and data distributions

Abstract

Federated Learning is a privacy-preserving decentralized approach for Machine Learning tasks. In industry deployments characterized by a limited number of entities possessing abundant data, the significance of a participant's role in shaping the global model becomes pivotal given that participation in a federation incurs costs, and participants may expect compensation for their involvement. Additionally, the contributions of participants serve as a crucial means to identify and address potential malicious actors and free-riders. However, fairly assessing individual contributions remains a significant hurdle. Recent works have demonstrated a considerable inherent instability in contribution estimations across aggregation strategies. While employing a different strategy may offer convergence benefits, this instability can have potentially harming effects on the willingness of participants in engaging in the federation. In this work, we introduce FedRandom, a novel mitigation technique to the contribution instability problem. Tackling the instability as a statistical estimation problem, FedRandom allows us to generate more samples than when using regular FL strategies. We show that these additional samples provide a more consistent and reliable evaluation of participant contributions. We demonstrate our approach using different data distributions across CIFAR-10, MNIST, CIFAR-100 and FMNIST and show that FedRandom reduces the overall distance to the ground truth by more than a third in half of all evaluated scenarios, and improves stability in more than 90% of cases.