Low-Rank Curvature for Zeroth-Order Optimization in LLM Fine-Tuning

TL;DR

LOREN is a curvature-aware zeroth-order optimization method that improves large language model fine-tuning by reducing variance and memory usage, outperforming existing methods in accuracy and convergence speed.

Contribution

We propose a novel curvature-aware ZO optimization approach that adaptively estimates anisotropic perturbations and captures curvature with a low-rank preconditioner, enhancing LLM fine-tuning.

Findings

Outperforms state-of-the-art ZO methods in accuracy and convergence.

Reduces peak memory usage by up to 27.3%.

Effectively captures curvature for better optimization.

Abstract

We introduce LOREN, a curvature-aware zeroth-order (ZO) optimization method for fine-tuning large language models (LLMs). Existing ZO methods, which estimate gradients via finite differences using random perturbations, often suffer from high variance and suboptimal search directions. Our approach addresses these challenges by: (i) reformulating the problem of gradient preconditioning as that of adaptively estimating an anisotropic perturbation distribution for gradient estimation, (ii) capturing curvature through a low-rank block diagonal preconditioner using the framework of natural evolution strategies, and (iii) applying a REINFORCE leave-one-out (RLOO) gradient estimator to reduce variance. Experiments on standard LLM benchmarks show that our method outperforms state-of-the-art ZO methods by achieving higher accuracy and faster convergence, while cutting peak memory usage by up to 27.3% compared with MeZO-Adam.