What Makes Local Updates Effective: The Role of Data Heterogeneity and Smoothness

TL;DR

This paper provides a theoretical analysis of local update algorithms like Local SGD, highlighting the importance of data heterogeneity and smoothness for their effectiveness in distributed and federated learning.

Contribution

It introduces a necessary and sufficient heterogeneity condition for local updates to outperform centralized methods and develops a new analysis framework with sharper convergence bounds.

Findings

Bounded second-order heterogeneity is key for local update success.

Established tight upper and lower bounds for various algorithms.

Provided regret bounds for online federated learning.

Abstract

This thesis contributes to the theoretical understanding of local update algorithms, especially Local SGD, in distributed and federated optimization under realistic models of data heterogeneity. A central focus is on the bounded second-order heterogeneity assumption, which is shown to be both necessary and sufficient for local updates to outperform centralized or mini-batch methods in convex and non-convex settings. The thesis establishes tight upper and lower bounds in several regimes for various local update algorithms and characterizes the min-max complexity of multiple problem classes. At its core is a fine-grained consensus-error-based analysis framework that yields sharper finite-time convergence bounds under third-order smoothness and relaxed heterogeneity assumptions. The thesis also extends to online federated learning, providing fundamental regret bounds under both first-order and bandit feedback. Together, these results clarify when and why local updates offer provable advantages, and the thesis serves as a self-contained guide for analyzing Local SGD in heterogeneous environments.