One-Shot Federated Learning: Theoretical Limits and Algorithms to Achieve Them

TL;DR

This paper establishes theoretical limits and proposes an optimal algorithm for one-shot federated learning under communication constraints, enabling accurate parameter estimation even with minimal per-machine data.

Contribution

It introduces the Multi-Resolution Estimator (MRE), achieving near-optimal error bounds in one-shot federated learning with communication limits, and analyzes performance in low-bit scenarios.

Findings

MRE achieves error bounds close to theoretical lower limits when B ≥ log(mn).

Expected error of MRE tends to zero as the number of machines increases, even with constant samples per machine.

The paper provides tight lower and upper bounds for learning under tiny communication budgets.

Abstract

We consider distributed statistical optimization in one-shot setting, where there are $m$ machines each observing $n$ i.i.d. samples. Based on its observed samples, each machine sends a $B$-bit-long message to a server. The server then collects messages from all machines, and estimates a parameter that minimizes an expected convex loss function. We investigate the impact of communication constraint, $B$, on the expected error and derive a tight lower bound on the error achievable by any algorithm. We then propose an estimator, which we call Multi-Resolution Estimator (MRE), whose expected error (when $B\ge\log mn$) meets the aforementioned lower bound up to poly-logarithmic factors, and is thereby order optimal. We also address the problem of learning under tiny communication budget, and present lower and upper error bounds when $B$ is a constant. The expected error of MRE, unlike existing algorithms, tends to zero as the number of machines ($m$) goes to infinity, even when the number of samples per machine ($n$) remains upper bounded by a constant. This property of the MRE algorithm makes it applicable in new machine learning paradigms where $m$ is much larger than $n$.