CLoQ: Enhancing Fine-Tuning of Quantized LLMs via Calibrated LoRA Initialization

TL;DR

This paper introduces CLoQ, a calibration-based initialization method for fine-tuning quantized large language models with LoRA, improving performance especially at ultra low-bit quantization levels.

Contribution

The paper proposes a novel calibration-based initialization strategy for LoRA in quantized LLMs, including a theoretical framework for optimal LoRA component construction.

Findings

CLoQ outperforms existing LoRA methods on multiple tasks.

Effective at ultra low-bit quantization levels.

Ensures strong fine-tuning foundation with minimal calibration data.

Abstract

Fine-tuning large language models (LLMs) using low-rank adaptation (LoRA) has become a highly efficient approach for downstream tasks, particularly in scenarios with limited computational resources. However, applying LoRA techniques to quantized LLMs poses unique challenges due to the reduced representational precision of quantized weights. In this paper, we introduce CLoQ (Calibrated LoRA initialization for Quantized LLMs), a simplistic initialization strategy designed to overcome these challenges. Our approach focuses on minimizing the layer-wise discrepancy between the original LLM and its quantized counterpart with LoRA components during initialization. By leveraging a small calibration dataset, CLoQ quantizes a pre-trained LLM and determines the optimal LoRA components for each layer, ensuring a strong foundation for subsequent fine-tuning. A key contribution of this work is a novel theoretical result that enables the accurate and closed-form construction of these optimal LoRA components. We validate the efficacy of CLoQ across multiple tasks such as language generation, arithmetic reasoning, and commonsense reasoning, demonstrating that it consistently outperforms existing LoRA fine-tuning methods for quantized LLMs, especially at ultra low-bit widths.