Effectiveness of Distributed Gradient Descent with Local Steps for Overparameterized Models

TL;DR

This paper analyzes the implicit bias of distributed Local Gradient Descent in overparameterized models, showing it converges to the centralized solution in direction and providing convergence rates, explaining its effectiveness with many local steps.

Contribution

It provides the first detailed analysis of the implicit bias of Local-GD in the interpolation regime, demonstrating convergence to the centralized model regardless of local steps count.

Findings

Local-GD converges to the centralized model in direction.

The convergence rate depends on the number of local steps.

Modified Local-GD achieves learning rate independence from local steps.

Abstract

In distributed training of machine learning models, gradient descent with local iterative steps, commonly known as Local (Stochastic) Gradient Descent (Local-(S)GD) or Federated averaging (FedAvg), is a very popular method to mitigate communication burden. In this method, gradient steps based on local datasets are taken independently in distributed compute nodes to update the local models, which are then aggregated intermittently. In the interpolation regime, Local-GD can converge to zero training loss. However, with many potential solutions corresponding to zero training loss, it is not known which solution Local-GD converges to. In this work we answer this question by analyzing implicit bias of Local-GD for classification tasks with linearly separable data. For the interpolation regime, our analysis shows that the aggregated global model obtained from Local-GD, with arbitrary number of local steps, converges exactly to the model that would be obtained if all data were in one place (centralized model) ''in direction''. Our result gives the exact rate of convergence to the centralized model with respect to the number of local steps. We also obtain the same implicit bias with a learning rate independent of number of local steps with a modified version of the Local-GD algorithm. Our analysis provides a new view to understand why Local-GD can still perform well with a very large number of local steps even for heterogeneous data. Lastly, we also discuss the extension of our results to Local-SGD and non-separable data.