Refined Analysis of Federated Averaging and Federated Richardson-Romberg

TL;DR

This paper offers a new analysis of Federated Averaging with constant step size, revealing its convergence to a stationary distribution and introducing a Richardson-Romberg extrapolation method to reduce bias caused by stochastic noise and client heterogeneity.

Contribution

It provides the first detailed bias analysis of FedAvg using Markov process theory and proposes a novel bias mitigation algorithm based on Richardson-Romberg extrapolation.

Findings

Convergence of FedAvg to a stationary distribution.

Bias decomposition into stochastic noise and heterogeneity components.

Introduction of a Richardson-Romberg based bias reduction method.

Abstract

In this paper, we present a novel analysis of \FedAvg with constant step size, relying on the Markov property of the underlying process. We demonstrate that the global iterates of the algorithm converge to a stationary distribution and analyze its resulting bias and variance relative to the problem's solution. We provide a first-order bias expansion in both homogeneous and heterogeneous settings. Interestingly, this bias decomposes into two distinct components: one that depends solely on stochastic gradient noise and another on client heterogeneity. Finally, we introduce a new algorithm based on the Richardson-Romberg extrapolation technique to mitigate this bias.