Preconditioned Federated Learning

TL;DR

This paper introduces adaptive preconditioned algorithms for federated learning that improve communication efficiency and convergence, demonstrating state-of-the-art results in diverse data settings.

Contribution

It proposes novel adaptive federated learning algorithms using covariance matrix preconditioning, with theoretical guarantees and superior empirical performance.

Findings

Achieves state-of-the-art performance on i.i.d. data

Effective in non-i.i.d. data settings

Provides convergence guarantees for the proposed algorithms

Abstract

Federated Learning (FL) is a distributed machine learning approach that enables model training in communication efficient and privacy-preserving manner. The standard optimization method in FL is Federated Averaging (FedAvg), which performs multiple local SGD steps between communication rounds. FedAvg has been considered to lack algorithm adaptivity compared to modern first-order adaptive optimizations. In this paper, we propose new communication-efficient FL algortithms based on two adaptive frameworks: local adaptivity (PreFed) and server-side adaptivity (PreFedOp). Proposed methods adopt adaptivity by using a novel covariance matrix preconditioner. Theoretically, we provide convergence guarantees for our algorithms. The empirical experiments show our methods achieve state-of-the-art performances on both i.i.d. and non-i.i.d. settings.