When to Trust Aggregated Gradients: Addressing Negative Client Sampling in Federated Learning

TL;DR

This paper investigates how client sampling impacts federated learning optimization and proposes an adaptive learning rate mechanism based on data distribution consistency to improve gradient reliability.

Contribution

It introduces a novel adaptive learning rate method that adjusts based on data distribution consistency, addressing negative effects of client sampling in federated learning.

Findings

Improved model convergence with adaptive learning rate.

Effective handling of non-i.i.d. data distributions.

Validated on image and text classification tasks.

Abstract

Federated Learning has become a widely-used framework which allows learning a global model on decentralized local datasets under the condition of protecting local data privacy. However, federated learning faces severe optimization difficulty when training samples are not independently and identically distributed (non-i.i.d.). In this paper, we point out that the client sampling practice plays a decisive role in the aforementioned optimization difficulty. We find that the negative client sampling will cause the merged data distribution of currently sampled clients heavily inconsistent with that of all available clients, and further make the aggregated gradient unreliable. To address this issue, we propose a novel learning rate adaptation mechanism to adaptively adjust the server learning rate for the aggregated gradient in each round, according to the consistency between the merged data distribution of currently sampled clients and that of all available clients. Specifically, we make theoretical deductions to find a meaningful and robust indicator that is positively related to the optimal server learning rate and can effectively reflect the merged data distribution of sampled clients, and we utilize it for the server learning rate adaptation. Extensive experiments on multiple image and text classification tasks validate the great effectiveness of our method.