On the Batch Size Selection in Stochastic Gradient Methods Using No-Replacement Sampling

TL;DR

This paper analyzes batch size selection in stochastic gradient methods using realistic sampling without replacement, showing it offers more accurate variance control and improved efficiency over traditional with-replacement approaches.

Contribution

It introduces a new batch size selection method based on sampling without replacement, with theoretical analysis and practical advantages over existing with-replacement methods.

Findings

Sampling without replacement better models practical variance reduction.

The new method ensures global convergence with smoother updates.

Improved efficiency in stochastic gradient methods.

Abstract

Recent stochastic gradient methods that have appeared in the literature base their efficiency and global convergence properties on a suitable control of the variance of the gradient batch estimate. This control is typically achieved by dynamically increasing the batch size during the iterations of the algorithm. However, in the existing methods the statistical analysis often relies on sampling with replacement. This particular batch selection appears unrealistic in practice. In this paper, we consider a more realistic approach to batch size selection based on sampling without replacement. The consequent statistical analysis is compared to that of sampling with replacement. The new batch size selection method, while still ensuring global convergence, provides a more accurate representation of the variance reduction observed in practice, leading to a smoother and more efficient batch size update scheme.