Mini-batch stochastic gradient descent with dynamic sample sizes

TL;DR

This paper introduces dynamic sample size rules for mini-batch stochastic gradient descent to improve convergence in constrained convex optimization, supported by empirical results showing superiority over fixed sample methods.

Contribution

It proposes novel dynamic sample size strategies that adaptively ensure descent directions with high probability in mini-batch SGD.

Findings

Superior convergence compared to fixed sample implementations

Effective in constrained convex optimization problems

Empirical validation on two applications

Abstract

We focus on solving constrained convex optimization problems using mini-batch stochastic gradient descent. Dynamic sample size rules are presented which ensure a descent direction with high probability. Empirical results from two applications show superior convergence compared to fixed sample implementations.