Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered

TL;DR

This paper argues that zeroth-order optimization in deep learning is underexplored rather than underpowered, highlighting its potential for scalable, resource-efficient training when rethought beyond traditional methods.

Contribution

It challenges prevailing assumptions about ZO methods' limitations and proposes new perspectives and opportunities for leveraging their unique advantages in deep learning.

Findings

ZO optimization can be scaled effectively with variance control techniques.

Spectral and subspace views enable interpretable variance reduction.

ZO's forward-only nature offers communication and resource efficiency advantages.

Abstract

Zeroth-order (ZO) optimization, learning from finite differences of function evaluations without backpropagation, has recently regained attention in deep learning due to its memory efficiency and applicability to gray- or black-box pipelines. Yet, ZO methods are often dismissed as fundamentally unscalable because of estimator variance and unfavorable query complexity. We argue that this conclusion might be misguided: ZO optimization is underexplored, not underpowered. We show that many perceived limitations stem from myopic development practices, most notably full-space, element-wise, estimator-centric designs. We articulate six positions spanning the algorithmic, systems, and evaluation stack. First, we revisit the feasibility boundaries of estimator-centric ZO methods through variance control, variance-query tradeoffs, and directional-derivative lenses. Then, we identify three underexplored opportunities: (i) subspace and spectral views of ZO that enable interpretable variance reduction with graceful query scaling, (ii) the forward-only nature of ZO as a systems advantage for communication-efficient, pipeline-friendly, and resource-constrained training, and (iii) the need to de-obfuscate ZO evaluations from task complexity. We strongly advocate rethinking ZO optimization around its unique strengths and acting accordingly, opening a viable path toward large-scale, system-aware, and resource-efficient learning with ZO optimization.