Boosting Resource-Constrained Federated Learning Systems with Guessed Updates

TL;DR

This paper introduces GEL, a novel guessing algorithm for federated learning that improves convergence speed and reduces tuning efforts in resource-constrained edge devices by enabling free gradientless updates.

Contribution

GEL is a flexible guessing method that enhances existing federated learning algorithms, significantly boosting convergence without extensive hyperparameter tuning.

Findings

GEL improves convergence by up to 40% in resource-constrained settings.

GEL reduces the need for exhaustive learning rate tuning.

GEL can be integrated with multiple state-of-the-art federated algorithms.

Abstract

Federated learning (FL) enables a set of client devices to collaboratively train a model without sharing raw data. This process, though, operates under the constrained computation and communication resources of edge devices. These constraints combined with systems heterogeneity force some participating clients to perform fewer local updates than expected by the server, thus slowing down convergence. Exhaustive tuning of hyperparameters in FL, furthermore, can be resource-intensive, without which the convergence is adversely affected. In this work, we propose GEL, the guess and learn algorithm. GEL enables constrained edge devices to perform additional learning through guessed updates on top of gradient-based steps. These guesses are gradientless, i.e., participating clients leverage them for free. Our generic guessing algorithm (i) can be flexibly combined with several state-of-the-art algorithms including FEDPROX, FEDNOVA, FEDYOGI or SCALEFL; and (ii) achieves significantly improved performance when the learning rates are not best tuned. We conduct extensive experiments and show that GEL can boost empirical convergence by up to 40% in resource constrained networks while relieving the need for exhaustive learning rate tuning.