LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits

TL;DR

LowRA is a novel framework that enables ultra-low-bit LoRA fine-tuning of large language models, significantly reducing memory usage while maintaining high performance, making it suitable for resource-constrained settings.

Contribution

LowRA introduces a new method for fine-tuning LLMs with less than 2 bits per parameter, combining optimized quantization and efficient kernels for scalable deployment.

Findings

Achieves accurate LoRA fine-tuning at 1.15 bits per parameter.

Reduces memory usage by up to 50%.

Maintains performance above 2 bits with minimal loss.

Abstract

Fine-tuning large language models (LLMs) is increasingly costly as models scale to hundreds of billions of parameters, and even parameter-efficient fine-tuning (PEFT) methods like LoRA remain resource-intensive. We introduce LowRA, the first framework to enable LoRA fine-tuning below 2 bits per parameter with minimal performance loss. LowRA optimizes fine-grained quantization - mapping, threshold selection, and precision assignment - while leveraging efficient CUDA kernels for scalable deployment. Extensive evaluations across 4 LLMs and 4 datasets show that LowRA achieves a superior performance-precision trade-off above 2 bits and remains accurate down to 1.15 bits, reducing memory usage by up to 50%. Our results highlight the potential of ultra-low-bit LoRA fine-tuning for resource-constrained environments.