ZOQO: Zero-Order Quantized Optimization

TL;DR

ZOQO is a novel zero-order optimization method that enables training quantized deep learning models efficiently, reducing computational demands while maintaining competitive performance in resource-constrained settings.

Contribution

It introduces a zero-order gradient approximation technique tailored for quantized models, facilitating training without full-precision gradients.

Findings

Achieves competitive accuracy with full-precision training.

Effective in fine-tuning large language models.

Applicable to black-box adversarial attacks.

Abstract

The increasing computational and memory demands in deep learning present significant challenges, especially in resource-constrained environments. We introduce a zero-order quantized optimization (ZOQO) method designed for training models with quantized parameters and operations. Our approach leverages zero-order approximations of the gradient sign and adapts the learning process to maintain the parameters' quantization without the need for full-precision gradient calculations. We demonstrate the effectiveness of ZOQO through experiments in fine-tuning of large language models and black-box adversarial attacks. Despite the limitations of zero-order and quantized operations training, our method achieves competitive performance compared to full-precision methods, highlighting its potential for low-resource environments.