Learning Dynamics of Zeroth-Order Optimization: A Kernel Perspective

TL;DR

This paper explains why zeroth-order optimization methods can effectively fine-tune large language models despite theoretical dimension-dependent slowdowns, by analyzing the empirical Neural Tangent Kernel and its dimension-free properties.

Contribution

The paper introduces a kernel perspective to understand the learning dynamics of zeroth-order methods, highlighting their scalability to large models through a dimension-free analysis.

Findings

The empirical NTK governs the learning behavior of ZO SGD.

The approximation error depends on output size, not parameter dimension.

This explains the scalability of ZO methods to large models like LLMs.

Abstract

Classical optimization theory establishes that zeroth-order (ZO) algorithms suffer from a dimension-dependent slowdown, with convergence rates typically scaling with the model dimension compared to first-order methods. However, in contrast to these theoretical expectations, a growing body of recent work demonstrates the successful application of ZO methods to fine-tuning Large Language Models (LLMs) with billions of parameters. To explain this paradox, we derive the one-step learning dynamics of ZO SGD, where the empirical Neural Tangent Kernel (eNTK) naturally emerges as the key term governing the learning behavior. Inspection of the eNTK produced by ZO SGD reveals that each element corresponds to the inner product of neural tangent vectors projected onto a random low-dimensional subspace. Thus, by invoking the Johnson-Lindenstrauss Lemma, our analysis shows that the fidelity of the ZO eNTK is governed primarily by the number of perturbations. Crucially, the approximation error depends on the model output size rather than the massive parameter dimension. This dimension-free property provides a theoretical justification for the scalability of ZO methods to LLMs finetuning tasks. We believe that this kernel-based framework offers a novel perspective for understanding ZO methods within the context of learning dynamics.