Zeroth-Order Optimization is Secretly Single-Step Policy Optimization

TL;DR

This paper reveals that Zeroth-Order Optimization methods are fundamentally equivalent to single-step Policy Optimization, providing new insights and practical variance reduction techniques that improve convergence and performance.

Contribution

It establishes a formal connection between ZOO and single-step PO, and introduces ZoAR, a novel ZOO algorithm with PO-inspired variance reduction methods.

Findings

ZOO with finite differences is equivalent to single-step PO.

REINFORCE gradient estimators are mathematically equivalent to ZOO estimators.

ZoAR outperforms existing methods in convergence speed and final performance.

Abstract

Zeroth-Order Optimization (ZOO) provides powerful tools for optimizing functions where explicit gradients are unavailable or expensive to compute. However, the underlying mechanisms of popular ZOO methods, particularly those employing randomized finite differences, and their connection to other optimization paradigms like Reinforcement Learning (RL) are not fully elucidated. This paper establishes a fundamental and previously unrecognized connection: ZOO with finite differences is equivalent to a specific instance of single-step Policy Optimization (PO). We formally unveil that the implicitly smoothed objective function optimized by common ZOO algorithms is identical to a single-step PO objective. Furthermore, we show that widely used ZOO gradient estimators, are mathematically equivalent to the REINFORCE gradient estimator with a specific baseline function, revealing the variance-reducing mechanism in ZOO from a PO perspective.Built on this unified framework, we propose ZoAR (Zeroth-Order Optimization with Averaged Baseline and Query Reuse), a novel ZOO algorithm incorporating PO-inspired variance reduction techniques: an averaged baseline from recent evaluations and query reuse analogous to experience replay. Our theoretical analysis further substantiates these techniques reduce variance and enhance convergence. Extensive empirical studies validate our theory and demonstrate that ZoAR significantly outperforms other methods in terms of convergence speed and final performance. Overall, our work provides a new theoretical lens for understanding ZOO and offers practical algorithmic improvements derived from its connection to PO.