A Markov Chain Theory Approach to Characterizing the Minimax Optimality of Stochastic Gradient Descent (for Least Squares)

TL;DR

This paper offers a simplified proof of the minimax optimality of averaged SGD for least squares by analyzing it as a stochastic process and characterizing its stationary covariance, including constant factors and model mis-specification.

Contribution

It provides a new, simplified proof of the minimax optimality of averaged SGD for least squares through stochastic process analysis and covariance characterization.

Findings

SGD is statistically minimax optimal for least squares.

Stationary covariance matrix of SGD is sharply characterized.

Results include constant factors and model mis-specification considerations.

Abstract

This work provides a simplified proof of the statistical minimax optimality of (iterate averaged) stochastic gradient descent (SGD), for the special case of least squares. This result is obtained by analyzing SGD as a stochastic process and by sharply characterizing the stationary covariance matrix of this process. The finite rate optimality characterization captures the constant factors and addresses model mis-specification.