Incremental Gradient Descent with Small Epoch Counts is Surprisingly Slow on Ill-Conditioned Problems

TL;DR

This paper investigates the convergence behavior of Incremental Gradient Descent (IGD) in the small epoch regime, revealing surprisingly slow convergence on ill-conditioned problems and highlighting the impact of component function assumptions.

Contribution

It provides the first lower bounds showing slow convergence of IGD in small epochs and analyzes how nonconvexity affects its optimality gap.

Findings

IGD can be surprisingly slow in small epoch regimes on ill-conditioned problems.

Nonconvex component functions can worsen IGD's optimality gap.

Tight bounds for IGD are established in the large epoch regime.

Abstract

Recent theoretical results demonstrate that the convergence rates of permutation-based SGD (e.g., random reshuffling SGD) are faster than uniform-sampling SGD; however, these studies focus mainly on the large epoch regime, where the number of epochs $K$ exceeds the condition number $\kappa$. In contrast, little is known when $K$ is smaller than $\kappa$, and it is still a challenging open question whether permutation-based SGD can converge faster in this small epoch regime (Safran and Shamir, 2021). As a step toward understanding this gap, we study the naive deterministic variant, Incremental Gradient Descent (IGD), on smooth and strongly convex functions. Our lower bounds reveal that for the small epoch regime, IGD can exhibit surprisingly slow convergence even when all component functions are strongly convex. Furthermore, when some component functions are allowed to be nonconvex, we prove that the optimality gap of IGD can be significantly worse throughout the small epoch regime. Our analyses reveal that the convergence properties of permutation-based SGD in the small epoch regime may vary drastically depending on the assumptions on component functions. Lastly, we supplement the paper with tight upper and lower bounds for IGD in the large epoch regime.