A Structured Tour of Optimization with Finite Differences

TL;DR

This paper reviews and extends structured finite-difference methods for zeroth-order optimization, showing they can match unstructured methods in cost while improving accuracy and convergence in various applications.

Contribution

It provides a comprehensive analysis of structured direction strategies, extending existing methods and empirically demonstrating their efficiency and effectiveness in optimization tasks.

Findings

Structured directions can be generated with similar computational costs as unstructured ones.

Structured approaches improve gradient estimation accuracy.

Enhanced convergence performance observed in real-world applications.

Abstract

Finite-difference methods are widely used for zeroth-order optimization in settings where gradient information is unavailable or expensive to compute. These procedures mimic first-order strategies by approximating gradients through function evaluations along a set of random directions. From a theoretical perspective, recent studies indicate that imposing structure (such as orthogonality) on the chosen directions allows for the derivation of convergence rates comparable to those achieved with unstructured random directions (i.e., directions sampled independently from a distribution). Empirically, although structured directions are expected to enhance performance, they often introduce additional computational costs, which can limit their applicability in high-dimensional settings. In this work, we examine the impact of structured direction selection in finite-difference methods. We review and extend several strategies for constructing structured direction matrices and compare them with unstructured approaches in terms of computational cost, gradient approximation quality, and convergence behavior. Our evaluation spans both synthetic tasks and real-world applications such as adversarial perturbation. The results demonstrate that structured directions can be generated with computational costs comparable to unstructured ones while significantly improving gradient estimation accuracy and optimization performance.