Batched Stochastic Gradient Descent with Weighted Sampling

TL;DR

This paper introduces a batched stochastic gradient descent method with weighted sampling that accelerates convergence for various objective functions, supported by theoretical analysis and experimental validation.

Contribution

It combines batching and weighted sampling in SGD, providing new schemes for optimal weights and demonstrating significant speedup over existing methods.

Findings

Speedup in convergence rate with batched weighted sampling

Efficient schemes for approximating optimal weights

Experimental validation showing substantial gains

Abstract

We analyze a batched variant of Stochastic Gradient Descent (SGD) with weighted sampling distribution for smooth and non-smooth objective functions. We show that by distributing the batches computationally, a significant speedup in the convergence rate is provably possible compared to either batched sampling or weighted sampling alone. We propose several computationally efficient schemes to approximate the optimal weights, and compute proposed sampling distributions explicitly for the least squares and hinge loss problems. We show both analytically and experimentally that substantial gains can be obtained.