Critical Bach Size Minimizes Stochastic First-Order Oracle Complexity of Deep Learning Optimizer using Hyperparameters Close to One

TL;DR

This paper demonstrates that there exists a critical batch size that minimizes the stochastic first-order oracle complexity in deep learning optimizers, and that using hyperparameters close to one with this batch size leads to faster convergence.

Contribution

The paper provides theoretical and empirical evidence for the existence of a critical batch size that minimizes SFO complexity and shows that optimizers like Adam perform optimally near this batch size with hyperparameters close to one.

Findings

Critical batch size minimizes SFO complexity.

Adam with hyperparameters near one converges faster.

SFO complexity increases beyond the critical batch size.

Abstract

Practical results have shown that deep learning optimizers using small constant learning rates, hyperparameters close to one, and large batch sizes can find the model parameters of deep neural networks that minimize the loss functions. We first show theoretical evidence that the momentum method (Momentum) and adaptive moment estimation (Adam) perform well in the sense that the upper bound of the theoretical performance measure is small with a small constant learning rate, hyperparameters close to one, and a large batch size. Next, we show that there exists a batch size called the critical batch size minimizing the stochastic first-order oracle (SFO) complexity, which is the stochastic gradient computation cost, and that SFO complexity increases once the batch size exceeds the critical batch size. Finally, we provide numerical results that support our theoretical results. That is, the numerical results indicate that Adam using a small constant learning rate, hyperparameters close to one, and the critical batch size minimizing SFO complexity has faster convergence than Momentum and stochastic gradient descent (SGD).