Optimized convergence of stochastic gradient descent by weighted averaging

TL;DR

This paper investigates weighted averaging strategies in stochastic gradient descent to improve convergence rates, especially in finite iterations, by balancing stochastic and optimization errors.

Contribution

It derives explicit formulas for errors and proposes parameter choices that reduce optimization error over standard averaging in stochastic gradient methods.

Findings

Weighted averaging can significantly reduce optimization error.

Parameter tuning balances stochastic and optimization errors effectively.

Numerical results support theoretical improvements and potential generalizations.

Abstract

Under mild assumptions stochastic gradient methods asymptotically achieve an optimal rate of convergence if the arithmetic mean of all iterates is returned as an approximate optimal solution. However, in the absence of stochastic noise, the arithmetic mean of all iterates converges considerably slower to the optimal solution than the iterates themselves. And also in the presence of noise, when a finite termination of the stochastic gradient method is considered, the arithmetic mean is not necessarily the best possible approximation to the unknown optimal solution. This paper aims at identifying optimal strategies in a particularly simple case, the minimization of a strongly convex function with i. i. d. noise terms and finite termination. Explicit formulas for the stochastic error and the optimization error are derived in dependence of certain parameters of the SGD method. The aim was to choose parameters such that both stochastic error and optimization error are reduced compared to arithmetic averaging. This aim could not be achieved; however, by allowing a slight increase of the stochastic error it was possible to select the parameters such that a significant reduction of the optimization error could be achieved. This reduction of the optimization error has a strong effect on the approximate solution generated by the stochastic gradient method in case that only a moderate number of iterations is used or when the initial error is large. The numerical examples confirm the theoretical results and suggest that a generalization to non-quadratic objective functions may be possible.