Local Updates in Distributed Optimization: Provable Acceleration and Topology Effects

TL;DR

This paper demonstrates that incorporating local updates into distributed optimization algorithms can accelerate convergence, with the extent of improvement depending on network topology and step size, supported by theoretical analysis and experiments.

Contribution

First rigorous proof that local updates can accelerate distributed optimization for a broad class of functions, highlighting optimal number of updates and network effects.

Findings

Two local updates suffice for maximal acceleration.

Speed gains depend on network spectral properties.

Experimental results confirm theoretical predictions.

Abstract

Inspired by the success of performing multiple local optimization steps between communication rounds in federated learning, incorporating such local updates into distributed optimization has recently attracted growing interest. However, unlike federated learning, where local updates can accelerate training by reducing gradient estimation error under minibatch settings, it remains unclear whether similar benefits persist when exact gradients are available. Moreover, existing theoretical results typically require reducing the step size when multiple local updates are employed, which can entirely offset any potential benefit of these additional local updates. In this paper, we focus on the classic DIGing algorithm and leverage the tight performance bounds provided by Performance Estimation Problems (PEP) to show that incorporating local updates can indeed accelerate distributed optimization. To the best of our knowledge, this is the first rigorous demonstration of such acceleration for a broad class of objective functions. Our analysis further reveals that, under an appropriate step size, performing only two local updates is sufficient to achieve the maximal possible improvement, and that additional local updates provide no further gains. Because more updates increase computational cost, these findings offer practical guidance for efficient implementation. We also show that these speed gains depend critically on the network structure, with sparser or less connected graphs, characterized by the spectral properties of the mixing matrix, yielding smaller improvements. Extensive experiments on both synthetic and real-world datasets corroborate the theoretical findings.