Constant Stepsize Local GD for Logistic Regression: Acceleration by Instability

TL;DR

This paper demonstrates that for logistic regression with heterogeneous data, using larger stepsizes in local gradient descent can accelerate convergence despite initial instability, surpassing traditional bounds.

Contribution

It provides a convergence analysis for local gradient descent with arbitrary stepsizes in heterogeneous logistic regression, revealing acceleration through instability.

Findings

Convergence rate of O(1/ηKR) after initial unstable phase

Initial instability lasts for approximately ηKM rounds

Accelerates beyond the standard O(1/R) rate for smooth convex objectives

Abstract

Existing analysis of Local (Stochastic) Gradient Descent for heterogeneous objectives requires stepsizes $\eta \leq 1/K$ where $K$ is the communication interval, which ensures monotonic decrease of the objective. In contrast, we analyze Local Gradient Descent for logistic regression with separable, heterogeneous data using any stepsize $\eta > 0$. With $R$ communication rounds and $M$ clients, we show convergence at a rate $\mathcal{O}(1/\eta K R)$ after an initial unstable phase lasting for $\widetilde{\mathcal{O}}(\eta K M)$ rounds. This improves upon the existing $\mathcal{O}(1/R)$ rate for general smooth, convex objectives. Our analysis parallels the single machine analysis of~\cite{wu2024large} in which instability is caused by extremely large stepsizes, but in our setting another source of instability is large local updates with heterogeneous objectives.