Local SGD in Overparameterized Linear Regression

TL;DR

This paper analyzes distributed stochastic gradient descent (SGD) in overparameterized linear regression, providing theoretical bounds, learning rates, and comparisons with distributed ridge regression to understand their relative efficiencies.

Contribution

It offers the first comprehensive theoretical analysis of local SGD in overparameterized linear regression, including bounds, rates, and comparisons with ridge regression.

Findings

Excess risk is proportional to variance when local nodes grow slowly.

Derived matching upper and lower bounds for learning rates.

Distributed SGD has smaller excess risk than distributed ridge regression under similar sample complexities.

Abstract

We consider distributed learning using constant stepsize SGD (DSGD) over several devices, each sending a final model update to a central server. In a final step, the local estimates are aggregated. We prove in the setting of overparameterized linear regression general upper bounds with matching lower bounds and derive learning rates for specific data generating distributions. We show that the excess risk is of order of the variance provided the number of local nodes grows not too large with the global sample size. We further compare the sample complexity of DSGD with the sample complexity of distributed ridge regression (DRR) and show that the excess SGD-risk is smaller than the excess RR-risk, where both sample complexities are of the same order.