Accelerating Decentralized Optimization via Overlapping Local Steps

TL;DR

OLDSGD is a novel decentralized optimization method that overlaps computation and communication to reduce idle time, maintaining convergence guarantees while significantly speeding up training in distributed learning environments.

Contribution

This paper introduces OLDSGD, a new approach that overlaps local computation with communication, improving efficiency without losing convergence properties in decentralized training.

Findings

OLDSGD reduces wall-clock time to convergence in experiments.

It maintains the same convergence rate as standard Local SGD.

OLDSGD is easy to implement with minimal modifications.

Abstract

Decentralized optimization has emerged as a critical paradigm for distributed learning, enabling scalable training while preserving data privacy through peer-to-peer collaboration. However, existing methods often suffer from communication bottlenecks due to frequent synchronization between nodes. We present Overlapping Local Decentralized SGD (OLDSGD), a novel approach to accelerate decentralized training by computation-communication overlapping, significantly reducing network idle time. With a deliberately designed update, OLDSGD preserves the same average update as Local SGD while avoiding communication-induced stalls. Theoretically, we establish non-asymptotic convergence rates for smooth non-convex objectives, showing that OLDSGD retains the same iteration complexity as standard Local Decentralized SGD while improving per-iteration runtime. Empirical results demonstrate OLDSGD's consistent improvements in wall-clock time convergence under different levels of communication delays. With minimal modifications to existing frameworks, OLDSGD offers a practical solution for faster decentralized learning without sacrificing theoretical guarantees.