On the equivalence of different adaptive batch size selection strategies for stochastic gradient descent methods

TL;DR

This paper shows the equivalence of different adaptive batch size strategies for SGD in terms of convergence, analyzes their computational costs, and illustrates the findings with two stochastic optimization problems.

Contribution

It proves the equivalence of the norm test and inner product test under certain conditions and compares their computational costs in adaptive batch size selection.

Findings

Norm test and inner product test are equivalent in convergence rates.

Inner product test can be as inexpensive as the norm test with optimal parameter choices.

Inner product test is never more computationally affordable than the norm test under the given condition.

Abstract

In this study, we demonstrate that the norm test and inner product/orthogonality test presented in \cite{Bol18} are equivalent in terms of the convergence rates associated with Stochastic Gradient Descent (SGD) methods if $\epsilon^2=\theta^2+\nu^2$ with specific choices of $\theta$ and $\nu$. Here, $\epsilon$ controls the relative statistical error of the norm of the gradient while $\theta$ and $\nu$ control the relative statistical error of the gradient in the direction of the gradient and in the direction orthogonal to the gradient, respectively. Furthermore, we demonstrate that the inner product/orthogonality test can be as inexpensive as the norm test in the best case scenario if $\theta$ and $\nu$ are optimally selected, but the inner product/orthogonality test will never be more computationally affordable than the norm test if $\epsilon^2=\theta^2+\nu^2$. Finally, we present two stochastic optimization problems to illustrate our results.