Towards Understanding the Generalizability of Delayed Stochastic Gradient Descent

TL;DR

This paper provides new theoretical bounds on the generalization error of asynchronous delayed stochastic gradient descent, showing that delays can actually reduce generalization error, supported by experiments.

Contribution

It introduces sharper generalization error bounds for delayed SGD using generating function analysis, revealing delays can improve generalization.

Findings

Asynchronous delays can reduce the generalization error of SGD.

New bounds are established for quadratic convex and strongly convex problems.

Experimental results support the theoretical findings.

Abstract

Stochastic gradient descent (SGD) performed in an asynchronous manner plays a crucial role in training large-scale machine learning models. However, the generalization performance of asynchronous delayed SGD, which is an essential metric for assessing machine learning algorithms, has rarely been explored. Existing generalization error bounds are rather pessimistic and cannot reveal the correlation between asynchronous delays and generalization. In this paper, we investigate sharper generalization error bound for SGD with asynchronous delay $\tau$. Leveraging the generating function analysis tool, we first establish the average stability of the delayed gradient algorithm. Based on this algorithmic stability, we provide upper bounds on the generalization error of $\tilde{\mathcal{O}}(\frac{T-\tau}{n\tau})$ and $\tilde{\mathcal{O}}(\frac{1}{n})$ for quadratic convex and strongly convex problems, respectively, where $T$ refers to the iteration number and $n$ is the amount of training data. Our theoretical results indicate that asynchronous delays reduce the generalization error of the delayed SGD algorithm. Analogous analysis can be generalized to the random delay setting, and the experimental results validate our theoretical findings.