ZOQO: Zero-Order Quantized Optimization
Noga Bar, Raja Giryes

TL;DR
ZOQO is a novel zero-order optimization method that enables training quantized deep learning models efficiently, reducing computational demands while maintaining competitive performance in resource-constrained settings.
Contribution
It introduces a zero-order gradient approximation technique tailored for quantized models, facilitating training without full-precision gradients.
Findings
Achieves competitive accuracy with full-precision training.
Effective in fine-tuning large language models.
Applicable to black-box adversarial attacks.
Abstract
The increasing computational and memory demands in deep learning present significant challenges, especially in resource-constrained environments. We introduce a zero-order quantized optimization (ZOQO) method designed for training models with quantized parameters and operations. Our approach leverages zero-order approximations of the gradient sign and adapts the learning process to maintain the parameters' quantization without the need for full-precision gradient calculations. We demonstrate the effectiveness of ZOQO through experiments in fine-tuning of large language models and black-box adversarial attacks. Despite the limitations of zero-order and quantized operations training, our method achieves competitive performance compared to full-precision methods, highlighting its potential for low-resource environments.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
