A Unified Convergence Analysis for Shuffling-Type Gradient Methods

TL;DR

This paper provides a comprehensive convergence analysis for various shuffling-type gradient methods in finite-sum optimization, covering both convex and nonconvex cases, with improved rates and practical insights.

Contribution

It introduces new convergence rates for shuffling gradient methods applicable to multiple sampling strategies, unifying and extending prior analyses.

Findings

New non-asymptotic and asymptotic convergence rates for nonconvex and convex settings.

Improved nonconvex convergence rate over existing methods.

Empirical validation on logistic regression and neural network training.

Abstract

In this paper, we propose a unified convergence analysis for a class of generic shuffling-type gradient methods for solving finite-sum optimization problems. Our analysis works with any sampling without replacement strategy and covers many known variants such as randomized reshuffling, deterministic or randomized single permutation, and cyclic and incremental gradient schemes. We focus on two different settings: strongly convex and nonconvex problems, but also discuss the non-strongly convex case. Our main contribution consists of new non-asymptotic and asymptotic convergence rates for a wide class of shuffling-type gradient methods in both nonconvex and convex settings. We also study uniformly randomized shuffling variants with different learning rates and model assumptions. While our rate in the nonconvex case is new and significantly improved over existing works under standard assumptions, the rate on the strongly convex one matches the existing best-known rates prior to this paper up to a constant factor without imposing a bounded gradient condition. Finally, we empirically illustrate our theoretical results via two numerical examples: nonconvex logistic regression and neural network training examples. As byproducts, our results suggest some appropriate choices for diminishing learning rates in certain shuffling variants.