The Power of Extrapolation in Federated Learning

TL;DR

This paper introduces server-extrapolation strategies for federated learning, specifically enhancing FedProx's convergence through theoretical analysis and novel extrapolation methods in convex settings.

Contribution

It proposes FedExProx with three extrapolation strategies and provides the first theoretical guarantees for such methods in federated learning.

Findings

Extrapolation improves convergence in federated learning.

Theoretical guarantees are established for the proposed methods.

Numerical experiments confirm the effectiveness of extrapolation strategies.

Abstract

We propose and study several server-extrapolation strategies for enhancing the theoretical and empirical convergence properties of the popular federated learning optimizer FedProx [Li et al., 2020]. While it has long been known that some form of extrapolation can help in the practice of FL, only a handful of works provide any theoretical guarantees. The phenomenon seems elusive, and our current theoretical understanding remains severely incomplete. In our work, we focus on smooth convex or strongly convex problems in the interpolation regime. In particular, we propose Extrapolated FedProx (FedExProx), and study three extrapolation strategies: a constant strategy (depending on various smoothness parameters and the number of participating devices), and two smoothness-adaptive strategies; one based on the notion of gradient diversity (FedExProx-GraDS), and the other one based on the stochastic Polyak stepsize (FedExProx-StoPS). Our theory is corroborated with carefully constructed numerical experiments.