Accelerating Stochastic Gradient Descent Using Antithetic Sampling

TL;DR

This paper introduces an antithetic sampling strategy for stochastic gradient descent that reduces variance by inducing negative correlation among gradients, leading to faster convergence in machine learning tasks.

Contribution

It proposes a novel variance reduction technique using antithetic sampling, which is practical and improves SGD efficiency without biasing the gradient estimate.

Findings

Variance of stochastic gradients is significantly reduced.

Faster convergence observed in binary classification tasks.

Antithetic sampling is computationally practical for real-world applications.

Abstract

(Mini-batch) Stochastic Gradient Descent is a popular optimization method which has been applied to many machine learning applications. But a rather high variance introduced by the stochastic gradient in each step may slow down the convergence. In this paper, we propose the antithetic sampling strategy to reduce the variance by taking advantage of the internal structure in dataset. Under this new strategy, stochastic gradients in a mini-batch are no longer independent but negatively correlated as much as possible, while the mini-batch stochastic gradient is still an unbiased estimator of full gradient. For the binary classification problems, we just need to calculate the antithetic samples in advance, and reuse the result in each iteration, which is practical. Experiments are provided to confirm the effectiveness of the proposed method.