FLRQ: Faster LLM Quantization with Flexible Low-Rank Matrix Sketching

TL;DR

FLRQ introduces a fast, flexible low-rank quantization method for large language models, significantly improving quantization quality and efficiency by adaptively selecting optimal ranks for each layer without costly fine-tuning.

Contribution

The paper proposes FLRQ, a novel low-rank quantization approach that quickly identifies optimal ranks for each layer, reducing computational overhead and enhancing model compression.

Findings

Achieves state-of-the-art quantization quality.

Demonstrates superior efficiency over existing methods.

Robust across diverse models and datasets.

Abstract

Traditional post-training quantization (PTQ) is considered an effective approach to reduce model size and accelerate inference of large-scale language models (LLMs). However, existing low-rank PTQ methods require costly fine-tuning to determine a compromise rank for diverse data and layers in large models, failing to exploit their full potential. Additionally, the current SVD-based low-rank approximation compounds the computational overhead. In this work, we thoroughly analyze the varying effectiveness of low-rank approximation across different layers in representative models. Accordingly, we introduce \underline{F}lexible \underline{L}ow-\underline{R}ank \underline{Q}uantization (FLRQ), a novel solution designed to quickly identify the accuracy-optimal ranks and aggregate them to achieve minimal storage combinations. FLRQ comprises two powerful components, Rank1-Sketch-based Flexible Rank Selection (R1-FLR) and Best Low-rank Approximation under Clipping (BLC). R1-FLR applies the R1-Sketch with Gaussian projection for the fast low-rank approximation, enabling outlier-aware rank extraction for each layer. Meanwhile, BLC aims at minimizing the low-rank quantization error under the scaling and clipping strategy through an iterative method. FLRQ demonstrates strong effectiveness and robustness in comprehensive experiments, achieving state-of-the-art performance in both quantization quality and algorithm efficiency.