Local Steps Speed Up Local GD for Heterogeneous Distributed Logistic Regression

TL;DR

This paper demonstrates that performing multiple local gradient steps in distributed logistic regression accelerates convergence, especially with large step sizes, improving over previous guarantees that showed no benefit from local updates.

Contribution

The paper provides the first convergence analysis showing the benefit of multiple local steps in heterogeneous distributed logistic regression with large step sizes.

Findings

Convergence rate of O(1/KR) for K local steps and R communication rounds.

Existing guarantees are at least Omega(1/R), showing no benefit from local steps.

Large step sizes (eta >> 1/K) enable faster progress in local gradient descent.

Abstract

We analyze two variants of Local Gradient Descent applied to distributed logistic regression with heterogeneous, separable data and show convergence at the rate $O(1/KR)$ for $K$ local steps and sufficiently large $R$ communication rounds. In contrast, all existing convergence guarantees for Local GD applied to any problem are at least $\Omega(1/R)$, meaning they fail to show the benefit of local updates. The key to our improved guarantee is showing progress on the logistic regression objective when using a large stepsize $\eta \gg 1/K$, whereas prior analysis depends on $\eta \leq 1/K$.