Federated Learning with Unbiased Gradient Aggregation and Controllable Meta Updating

TL;DR

This paper introduces unbiased gradient aggregation and controllable meta updating techniques to improve federated learning, resulting in faster convergence and higher accuracy across diverse models and settings.

Contribution

It proposes novel unbiased gradient aggregation and meta updating methods that enhance convergence speed and accuracy in federated learning.

Findings

Faster convergence compared to traditional FedAvg.

Higher accuracy across various network architectures.

Applicable to different models and federated learning scenarios.

Abstract

Federated learning (FL) aims to train machine learning models in the decentralized system consisting of an enormous amount of smart edge devices. Federated averaging (FedAvg), the fundamental algorithm in FL settings, proposes on-device training and model aggregation to avoid the potential heavy communication costs and privacy concerns brought by transmitting raw data. However, through theoretical analysis we argue that 1) the multiple steps of local updating will result in gradient biases and 2) there is an inconsistency between the expected target distribution and the optimization objectives following the training paradigm in FedAvg. To tackle these problems, we first propose an unbiased gradient aggregation algorithm with the keep-trace gradient descent and the gradient evaluation strategy. Then we introduce an additional controllable meta updating procedure with a small set of data samples, indicating the expected target distribution, to provide a clear and consistent optimization objective. Both the two improvements are model- and task-agnostic and can be applied individually or together. Experimental results demonstrate that the proposed methods are faster in convergence and achieve higher accuracy with different network architectures in various FL settings.