Eigen-componentwise convergence of SGD on quadratic programming

TL;DR

This paper analyzes how stochastic gradient descent (SGD) converges on quadratic programming problems, revealing that convergence rates vary across eigencomponents and depend on eigenvalues, with a phase transition affecting convergence speed.

Contribution

It provides a detailed eigencomponentwise analysis of SGD convergence on quadratic problems, highlighting phase transitions and the impact of eigenvalues on convergence speed.

Findings

Eigencomponents with larger eigenvalues converge faster initially.

A phase transition causes convergence speed to decrease for certain components.

Overall error decay is faster initially and slows down asymptotically.

Abstract

Stochastic gradient descent (SGD) is a workhorse algorithm for solving large-scale optimization problems in data science and machine learning. Understanding the convergence of SGD is hence of fundamental importance. In this work we examine the SGD convergence (with various step sizes) when applied to unconstrained convex quadratic programming (essentially least-squares (LS) problems), and in particular analyze the error components respect to the eigenvectors of the Hessian. The main message is that the convergence depends largely on the corresponding eigenvalues (singular values of the coefficient matrix in the LS context), namely the components for the large singular values converge faster in the initial phase. We then show there is a phase transition in the convergence where the convergence speed of the components, especially those corresponding to the larger singular values, will decrease. Finally, we show that the convergence of the overall error (in the solution) tends to decay as more iterations are run, that is, the initial convergence is faster than the asymptote.