Multi-index Antithetic Stochastic Gradient Algorithm

TL;DR

This paper introduces MASGA, a novel stochastic gradient algorithm that reduces variance without relying on distribution structure, achieving optimal mean square error performance in Monte Carlo estimation.

Contribution

The paper develops MASGA, an adaptive multi-index antithetic stochastic gradient algorithm that is structure-independent and optimal in mean square error versus computational cost.

Findings

MASGA matches Monte Carlo estimators with unbiased samples in log-concave settings.

Numerical experiments confirm MASGA's effectiveness beyond log-concave cases.

Theoretical proof of optimality in mean square error for log-concave distributions.

Abstract

Stochastic Gradient Algorithms (SGAs) are ubiquitous in computational statistics, machine learning and optimisation. Recent years have brought an influx of interest in SGAs, and the non-asymptotic analysis of their bias is by now well-developed. However, relatively little is known about the optimal choice of the random approximation (e.g mini-batching) of the gradient in SGAs as this relies on the analysis of the variance and is problem specific. While there have been numerous attempts to reduce the variance of SGAs, these typically exploit a particular structure of the sampled distribution by requiring a priori knowledge of its density's mode. It is thus unclear how to adapt such algorithms to non-log-concave settings. In this paper, we construct a Multi-index Antithetic Stochastic Gradient Algorithm (MASGA) whose implementation is independent of the structure of the target measure and which achieves performance on par with Monte Carlo estimators that have access to unbiased samples from the distribution of interest. In other words, MASGA is an optimal estimator from the mean square error-computational cost perspective within the class of Monte Carlo estimators. We prove this fact rigorously for log-concave settings and verify it numerically for some examples where the log-concavity assumption is not satisfied.