FedSplit: An algorithmic framework for fast federated optimization

TL;DR

FedSplit introduces a new federated optimization framework based on operator splitting, ensuring convergence to true optima and robustness to inexact computations, with promising practical benefits.

Contribution

The paper proposes FedSplit, a novel operator splitting-based algorithm for federated optimization that guarantees convergence to the true solution and handles inexact local computations.

Findings

FedSplit converges to the true optima in convex federated problems.

The method is robust to inexact local computations.

Experiments show practical advantages of FedSplit.

Abstract

Motivated by federated learning, we consider the hub-and-spoke model of distributed optimization in which a central authority coordinates the computation of a solution among many agents while limiting communication. We first study some past procedures for federated optimization, and show that their fixed points need not correspond to stationary points of the original optimization problem, even in simple convex settings with deterministic updates. In order to remedy these issues, we introduce FedSplit, a class of algorithms based on operator splitting procedures for solving distributed convex minimization with additive structure. We prove that these procedures have the correct fixed points, corresponding to optima of the original optimization problem, and we characterize their convergence rates under different settings. Our theory shows that these methods are provably robust to inexact computation of intermediate local quantities. We complement our theory with some simple experiments that demonstrate the benefits of our methods in practice.