Kron-LoRA: Hybrid Kronecker-LoRA Adapters for Scalable, Sustainable Fine-tuning

TL;DR

Kron-LoRA is a hybrid adapter method combining Kronecker and low-rank techniques, enabling scalable, parameter-efficient fine-tuning of large language models with comparable or better performance than existing methods.

Contribution

It introduces Kron-LoRA, a novel hybrid adapter that reduces parameters significantly while maintaining expressivity, advancing scalable fine-tuning of large models.

Findings

Achieves up to 4x fewer parameters than standard LoRA.

Matches or exceeds LoRA performance across multiple benchmarks.

Maintains competitive transferability with only a quarter of the parameters.

Abstract

Fine-tuning massive pre-trained language models across many tasks demands adapters that are both parameter-efficient and expressive. We introduce \textbf{Kron-LoRA}, a hybrid adapter that combines Kronecker-structured factorization with low-rank LoRA compression-an integration that, to our knowledge, has not been explored in parameter-efficient fine-tuning or in matrix approximation literature. Kron-LoRA achieves up to 4$\times$ fewer parameters than standard LoRA while retaining similar expressivity. Experiments on DistilBERT, Mistral-7B, LLaMA-2-7B, and LLaMA-3-8B across eight benchmarks show that Kron-LoRA matches or exceeds LoRA baselines with modest memory savings and only a 5-8\% speed overhead. In sequential fine-tuning, it also delivers competitive cross-task transfer despite using only one-quarter of the adapter parameters. Kron-LoRA thus offers a scalable, sustainable solution for multi-task adaptation of large language models.