Towards Fast LLM Fine-tuning through Zeroth-Order Optimization with Projected Gradient-Aligned Perturbations

TL;DR

This paper introduces P-GAP, a zeroth-order optimization method with gradient alignment that significantly accelerates large language model fine-tuning, reducing training time and resource usage while improving accuracy.

Contribution

P-GAP is a novel zeroth-order optimization technique that estimates a low-dimensional gradient space and aligns perturbations, leading to faster and more efficient LLM fine-tuning.

Findings

Up to 6% accuracy improvement on classification tasks

Up to 12% higher accuracy on generation tasks

Reduced training iterations and GPU hours by over 70%

Abstract

Fine-tuning large language models (LLMs) using zeroth-order (ZO) optimization has emerged as a promising alternative to traditional gradient-based methods due to its reduced memory footprint requirement. However, existing ZO methods suffer from high variance in gradient estimation, leading to slow convergence and suboptimal performance on large-scale models. In this work, we propose P-GAP, a fast LLM fine-tuning approach through zeroth-order optimization with Projected Gradient-Aligned Perturbations. Specifically, we first estimate a low-dimensional gradient space and then align perturbations in projected gradients' direction within the space. This approach enables reduced the number of perturbed parameters and decreased variance, therefore accelerated convergence for LLM fine-tuning. Experiments on LLMs show that P-GAP consistently surpasses the baselines, achieving up to 6% increase in accuracy on classification tasks and up to 12% higher accuracy on generation tasks, with up to about 81% less training iterations and 70% less GPU hours. These results demonstrate that P-GAP enables fast, scalable, and resource-efficient ZO LLM fine-tuning.