Why Random Reshuffling Beats Stochastic Gradient Descent

TL;DR

This paper provides a rigorous convergence analysis of the random reshuffling (RR) method, demonstrating its superior convergence rate over stochastic gradient descent (SGD) for strongly convex functions, and introduces a modified RR with even faster convergence.

Contribution

It offers the first theoretical convergence rate analysis of RR, showing it outperforms SGD, and proposes a modified RR with accelerated convergence rate.

Findings

RR with averaging converges at rate Θ(1/k^{2s}) for smooth strongly convex functions.

RR outperforms SGD with a convergence rate Ω(1/k).

A modified RR achieves a convergence rate of O(1/k^2).

Abstract

We analyze the convergence rate of the random reshuffling (RR) method, which is a randomized first-order incremental algorithm for minimizing a finite sum of convex component functions. RR proceeds in cycles, picking a uniformly random order (permutation) and processing the component functions one at a time according to this order, i.e., at each cycle, each component function is sampled without replacement from the collection. Though RR has been numerically observed to outperform its with-replacement counterpart stochastic gradient descent (SGD), characterization of its convergence rate has been a long standing open question. In this paper, we answer this question by showing that when the component functions are quadratics or smooth and the sum function is strongly convex, RR with iterate averaging and a diminishing stepsize $\alpha_k=\Theta(1/k^s)$ for $s\in (1/2,1)$ converges at rate $\Theta(1/k^{2s})$ with probability one in the suboptimality of the objective value, thus improving upon the $\Omega(1/k)$ rate of SGD. Our analysis draws on the theory of Polyak-Ruppert averaging and relies on decoupling the dependent cycle gradient error into an independent term over cycles and another term dominated by $\alpha_k^2$. This allows us to apply law of large numbers to an appropriately weighted version of the cycle gradient errors, where the weights depend on the stepsize. We also provide high probability convergence rate estimates that shows decay rate of different terms and allows us to propose a modification of RR with convergence rate ${\cal O}(\frac{1}{k^2})$.