A Rule for Gradient Estimator Selection, with an Application to Variational Inference

TL;DR

This paper presents a simple rule for selecting the optimal gradient estimator in stochastic gradient descent to improve convergence, applicable to various SGD variants and objective assumptions, with an automatic selection technique demonstrated empirically.

Contribution

It introduces a convergence-based rule for gradient estimator selection and develops an automatic selection method for finite and infinite estimator pools.

Findings

Automatic estimator selection performs comparably to hindsight-optimal choices.

The rule applies across different SGD variants and objective assumptions.

The approach is validated through empirical experiments.

Abstract

Stochastic gradient descent (SGD) is the workhorse of modern machine learning. Sometimes, there are many different potential gradient estimators that can be used. When so, choosing the one with the best tradeoff between cost and variance is important. This paper analyzes the convergence rates of SGD as a function of time, rather than iterations. This results in a simple rule to select the estimator that leads to the best optimization convergence guarantee. This choice is the same for different variants of SGD, and with different assumptions about the objective (e.g. convexity or smoothness). Inspired by this principle, we propose a technique to automatically select an estimator when a finite pool of estimators is given. Then, we extend to infinite pools of estimators, where each one is indexed by control variate weights. This is enabled by a reduction to a mixed-integer quadratic program. Empirically, automatically choosing an estimator performs comparably to the best estimator chosen with hindsight.