A Correlation-induced Finite Difference Estimator

TL;DR

This paper introduces a novel correlation-based finite difference estimator that reduces variance and bias in stochastic gradient estimation, especially effective in small sample and high-dimensional derivative-free optimization scenarios.

Contribution

It presents a sample-driven bootstrap method for optimal perturbation estimation and a correlated sample FD estimator that improves variance and bias over traditional methods.

Findings

Reduces variance and bias compared to traditional FD estimators.

Effective in small sample size scenarios.

Successfully applied to high-dimensional derivative-free optimization.

Abstract

Finite difference (FD) approximation is a classic approach to stochastic gradient estimation when only noisy function realizations are available. In this paper, we first provide a sample-driven method via the bootstrap technique to estimate the optimal perturbation, and then propose an efficient FD estimator based on correlated samples at the estimated optimal perturbation. Furthermore, theoretical analyses of both the perturbation estimator and the FD estimator reveal that, {\it surprisingly}, the correlation enables the proposed FD estimator to achieve a reduction in variance and, in some cases, a decrease in bias compared to the traditional optimal FD estimator. Numerical results confirm the efficiency of our estimators and align well with the theory presented, especially in scenarios with small sample sizes. Finally, we apply the estimator to solve derivative-free optimization (DFO) problems, and numerical studies show that DFO problems with 100 dimensions can be effectively solved.