QET: Enhancing Quantized LLM Parameters and KV cache Compression through Element Substitution and Residual Clustering

TL;DR

This paper introduces QET, a novel method for matrix quantization that minimizes error and enhances compression efficiency in large language models and cache systems through element substitution and residual clustering.

Contribution

We formulate the QEM problem for matrix quantization, design the QET algorithm leveraging local element orderliness, and propose optimizations for improved accuracy and speed.

Findings

QET reduces MSE to 5.05% of the best existing method on LLM datasets.

QET achieves significant error reduction on K cache and V cache.

Optimizations improve both quantization accuracy and computational efficiency.

Abstract

The matrix quantization entails representing matrix elements in a more space-efficient form to reduce storage usage, with dequantization restoring the original matrix for use. We formulate the Quantization Error Minimization (QEM) problem as minimizing the distance between a matrix before and after quantization, under the condition that the quantized matrix occupies the same memory space. Matrix quantization is crucial in various applications, including Large Language Models (LLMs) weight quantization, vector databases, KV cache quantization, graph compression, and image compression. Recent advancements in LLMs, such as GPT-4 and BERT, have highlighted the importance of matrix compression due to the large size of parameters and KV cache, which are stored as matrices. We propose Quantum Entanglement Trees (QET) to address the QEM problem by leveraging the local orderliness of matrix elements, involving iterative element swapping to form a locally ordered matrix. This matrix is then grouped and quantized by columns. To enhance QET, we introduce two optimizations: further quantizing residuals to reduce MSE, and using masking and batch processing to accelerate the algorithm. Experimental results demonstrate that QET can effectively reduce MSE to 5.05%, 13.33%, and 11.89% of the current best method on the LLM dataset, K cache, and V cache, respectively. Our contributions include the abstraction of the QEM problem, the design of the QET algorithm, and the proposal of two optimizations to improve accuracy and speed.