QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning

TL;DR

QDyLoRA introduces a dynamic, quantized low-rank adaptation method that allows efficient fine-tuning of large language models across multiple ranks on limited hardware, outperforming existing static approaches.

Contribution

It proposes a novel quantized dynamic low-rank adaptation technique enabling flexible, efficient LLM fine-tuning across various ranks without additional training.

Findings

QDyLoRA can fine-tune Falcon-40b across ranks 1 to 64 on a single GPU.

QDyLoRA is competitive with QLoRA and outperforms at optimal ranks.

The method reduces hardware requirements for large model fine-tuning.

Abstract

Finetuning large language models requires huge GPU memory, restricting the choice to acquire Larger models. While the quantized version of the Low-Rank Adaptation technique, named QLoRA, significantly alleviates this issue, finding the efficient LoRA rank is still challenging. Moreover, QLoRA is trained on a pre-defined rank and, therefore, cannot be reconfigured for its lower ranks without requiring further fine-tuning steps. This paper proposes QDyLoRA -Quantized Dynamic Low-Rank Adaptation-, as an efficient quantization approach for dynamic low-rank adaptation. Motivated by Dynamic LoRA, QDyLoRA is able to efficiently finetune LLMs on a set of pre-defined LoRA ranks. QDyLoRA enables fine-tuning Falcon-40b for ranks 1 to 64 on a single 32 GB V100-GPU through one round of fine-tuning. Experimental results show that QDyLoRA is competitive to QLoRA and outperforms when employing its optimal rank.