How Good is SGD with Random Shuffling?

TL;DR

This paper analyzes the theoretical performance limits of SGD with random shuffling in finite-sum optimization, revealing a fundamental gap between single and multiple reshuffling strategies and establishing lower bounds that match upper bounds in specific cases.

Contribution

It provides the first lower bounds on the expected error of SGD with random reshuffling, clarifying the advantages and limitations of different shuffling heuristics.

Findings

Lower bounds show inherent performance gaps between single and repeated reshuffling.

Re-shuffling after each pass yields better error rates than a single shuffle.

Matching upper bounds are established for univariate quadratic functions.

Abstract

We study the performance of stochastic gradient descent (SGD) on smooth and strongly-convex finite-sum optimization problems. In contrast to the majority of existing theoretical works, which assume that individual functions are sampled with replacement, we focus here on popular but poorly-understood heuristics, which involve going over random permutations of the individual functions. This setting has been investigated in several recent works, but the optimal error rates remain unclear. In this paper, we provide lower bounds on the expected optimization error with these heuristics (using SGD with any constant step size), which elucidate their advantages and disadvantages. In particular, we prove that after $k$ passes over $n$ individual functions, if the functions are re-shuffled after every pass, the best possible optimization error for SGD is at least $\Omega\left(1/(nk)^2+1/nk^3\right)$, which partially corresponds to recently derived upper bounds. Moreover, if the functions are only shuffled once, then the lower bound increases to $\Omega(1/nk^2)$. Since there are strictly smaller upper bounds for repeated reshuffling, this proves an inherent performance gap between SGD with single shuffling and repeated shuffling. As a more minor contribution, we also provide a non-asymptotic $\Omega(1/k^2)$ lower bound (independent of $n$) for the incremental gradient method, when no random shuffling takes place. Finally, we provide an indication that our lower bounds are tight, by proving matching upper bounds for univariate quadratic functions.