Increasing Batch Size Improves Convergence of Stochastic Gradient Descent with Momentum

TL;DR

This paper demonstrates that increasing batch size during training with stochastic gradient descent with momentum enhances convergence speed and reduces computational costs, supported by theoretical analysis and numerical experiments.

Contribution

It provides a theoretical and empirical analysis showing that increasing batch size improves convergence of mini-batch SGDM in deep neural network training.

Findings

Increasing batch size minimizes the expectation of the full gradient norm.

An increasing batch size converges faster to stationary points than a constant batch size.

Using increasing batch size reduces computational costs during training.

Abstract

Stochastic gradient descent with momentum (SGDM), in which a momentum term is added to SGD, has been well studied in both theory and practice. The theoretical studies show that the settings of the learning rate and momentum weight affect the convergence of SGDM. Meanwhile, the practical studies have shown that the batch-size setting strongly affects the performance of SGDM. In this paper, we focus on mini-batch SGDM with a constant learning rate and constant momentum weight, which is frequently used to train deep neural networks. We show theoretically that using a constant batch size does not always minimize the expectation of the full gradient norm of the empirical loss in training a deep neural network, whereas using an increasing batch size definitely minimizes it; that is, an increasing batch size improves the convergence of mini-batch SGDM. We also provide numerical results supporting our analyses, indicating specifically that mini-batch SGDM with an increasing batch size converges to stationary points faster than with a constant batch size, while also reducing computational cost. Python implementations of the optimizers used in the numerical experiments are available at https://github.com/iiduka-researches/NSHB_increasing_batchsize_acml25/.