Quantized Evolution Strategies: High-precision Fine-tuning of Quantized LLMs at Low-precision Cost

TL;DR

This paper introduces Quantized Evolution Strategies (QES), a novel method enabling high-precision fine-tuning of quantized large language models directly in their discrete parameter space, overcoming previous limitations of gradient-based methods.

Contribution

QES is the first approach to perform full-parameter fine-tuning in the quantized space using a backpropagation-free optimization with error feedback and seed replay, enabling scalable LLM fine-tuning.

Findings

QES outperforms state-of-the-art zeroth-order fine-tuning methods.

QES enables effective fine-tuning of quantized models for arithmetic reasoning.

QES reduces memory usage to low-precision inference levels.

Abstract

Post-Training Quantization (PTQ) is essential for deploying Large Language Models (LLMs) on memory-constrained devices, yet it renders models static and difficult to fine-tune. Standard fine-tuning paradigms, including Reinforcement Learning (RL), fundamentally rely on backpropagation and high-precision weights to compute gradients. Thus they cannot be used on quantized models, where the parameter space is discrete and non-differentiable. While Evolution Strategies (ES) offer a backpropagation-free alternative, optimization of the quantized parameters can still fail due to vanishing or inaccurate gradient. This paper introduces Quantized Evolution Strategies (QES), an optimization paradigm that performs full-parameter fine-tuning directly in the quantized space. QES is based on two innovations: (1) it integrates accumulated error feedback to preserve high-precision gradient signals, and (2) it utilizes a stateless seed replay to reduce memory usage to low-precision inference levels. QES significantly outperforms the state-of-the-art zeroth-order fine-tuning method on arithmetic reasoning tasks, making direct fine-tuning for quantized models possible. It therefore opens up the possibility for scaling up LLMs entirely in the quantized space. The source code is available at https://github.com/dibbla/Quantized-Evolution-Strategies .