Local Exact-Diffusion for Decentralized Optimization and Learning

TL;DR

This paper introduces Local Exact-Diffusion (LED), a decentralized optimization method with local updates that improves convergence rates and reduces communication costs in distributed learning, applicable to both convex and nonconvex problems.

Contribution

The paper proposes LED, a novel decentralized local update algorithm with proven convergence and improved rates, connecting it to existing methods and demonstrating its effectiveness.

Findings

LED achieves faster convergence than existing decentralized methods.

The method generalizes to centralized networks, matching state-of-the-art bounds.

Numerical experiments confirm the benefits of local updates in decentralized settings.

Abstract

Distributed optimization methods with local updates have recently attracted a lot of attention due to their potential to reduce the communication cost of distributed methods. In these algorithms, a collection of nodes performs several local updates based on their local data, and then they communicate with each other to exchange estimate information. While there have been many studies on distributed local methods with centralized network connections, there has been less work on decentralized networks. In this work, we propose and investigate a locally updated decentralized method called Local Exact-Diffusion (LED). We establish the convergence of LED in both convex and nonconvex settings for the stochastic online setting. Our convergence rate improves over the rate of existing decentralized methods. When we specialize the network to the centralized case, we recover the state-of-the-art bound for centralized methods. We also link LED to several other independently studied distributed methods, including Scaffnew, FedGate, and VRL-SGD. Additionally, we numerically investigate the benefits of local updates for decentralized networks and demonstrate the effectiveness of the proposed method.