Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptation

TL;DR

This paper introduces LoRDS, a low-rank decomposition framework for LLM quantization and adaptation that surpasses block-wise methods in efficiency and expressiveness, enabling high-fidelity quantization and high-rank PEFT without additional inference costs.

Contribution

LoRDS rethinks quantization granularity by modeling scaling as continuous low-rank matrices, unifying quantization and adaptation with superior performance and efficiency.

Findings

Outperforms state-of-the-art baselines in quantization and fine-tuning.

Achieves up to 27% accuracy improvement at 3 bits on Llama3-8B.

Provides 1.5x inference speedup on NVIDIA RTX 4090.

Abstract

Current quantization methods for LLMs predominantly rely on block-wise structures to maintain efficiency, often at the cost of representational flexibility. In this work, we demonstrate that element-wise quantization can be made as efficient as block-wise scaling while providing strictly superior expressive power by modeling the scaling manifold as continuous low-rank matrices ($S = BA$). We propose Low-Rank Decomposed Scaling (LoRDS), a unified framework that rethinks quantization granularity through this low-rank decomposition. By "breaking the blocks" of spatial constraints, LoRDS establishes a seamless efficiency lifecycle: it provides high-fidelity PTQ initialization refined via iterative optimization, enables joint QAT of weights and scaling factors, and facilitates high-rank multiplicative PEFT adaptation. Unlike additive PEFT approaches such as QLoRA, LoRDS enables high-rank weight updates within a low-rank budget while incurring no additional inference overhead. Supported by highly optimized Triton kernels, LoRDS consistently outperforms state-of-the-art baselines across various model families in both quantization and downstream fine-tuning tasks. Notably, on Llama3-8B, our method achieves up to a 27.0% accuracy improvement at 3 bits over NormalFloat quantization and delivers a 1.5x inference speedup on NVIDIA RTX 4090 while enhancing PEFT performance by 9.6% on downstream tasks over 4bit QLoRA, offering a robust and integrated solution for unified compression and adaptation of LLMs.