Minimax Efficient Finite-Difference Stochastic Gradient Estimators Using Black-Box Function Evaluations

TL;DR

This paper demonstrates that central finite-difference methods are nearly minimax optimal for stochastic gradient estimation from noisy black-box function evaluations, outperforming other estimators in terms of statistical accuracy.

Contribution

It establishes the near-optimality of central finite-difference estimators within a minimax framework for zeroth-order stochastic gradient estimation.

Findings

Central finite-difference is nearly minimax optimal.

Optimality holds among linear and nonlinear estimators.

Improves understanding of the best possible accuracy for black-box gradient estimation.

Abstract

Standard approaches to stochastic gradient estimation, with only noisy black-box function evaluations, use the finite-difference method or its variants. While natural, it is open to our knowledge whether their statistical accuracy is the best possible. This paper argues so by showing that central finite-difference is a nearly minimax optimal zeroth-order gradient estimator for a suitable class of objective functions and mean squared risk, among both the class of linear estimators and the much larger class of all (nonlinear) estimators.