CPTQuant - A Novel Mixed Precision Post-Training Quantization Techniques for Large Language Models

TL;DR

CPTQuant introduces innovative mixed precision post-training quantization methods that significantly reduce large language models' size and computational demands while maintaining accuracy, through correlation, pruning, and Taylor decomposition techniques.

Contribution

The paper presents CPTQuant, a comprehensive framework combining three novel mixed precision quantization strategies tailored for large language models, enhancing compression and efficiency.

Findings

Up to 4x model compression with minimal accuracy loss.

PMPQ achieves higher compression ratios than existing methods.

TDMPQ attains 30% greater compression for language modeling tasks.

Abstract

Large language models have transformed the comprehension and generation of natural language tasks, but they come with substantial memory and computational requirements. Quantization techniques have emerged as a promising avenue for addressing these challenges while preserving accuracy and making energy efficient. We propose CPTQuant, a comprehensive strategy that introduces correlation-based (CMPQ), pruning-based (PMPQ), and Taylor decomposition-based (TDMPQ) mixed precision techniques. CMPQ adapts the precision level based on canonical correlation analysis of different layers. PMPQ optimizes precision layer-wise based on their sensitivity to sparsity. TDMPQ modifies precision using Taylor decomposition to assess each layer's sensitivity to input perturbation. These strategies allocate higher precision to more sensitive layers while diminishing precision to robust layers. CPTQuant assesses the performance across BERT, OPT-125M, OPT-350M, OPT-1.3B, and OPT-2.7B. We demonstrate up to 4x compression and a 2x-fold increase in efficiency with minimal accuracy drop compared to Hugging Face FP16. PMPQ stands out for achieving a considerably higher model compression. Sensitivity analyses across various LLMs show that the initial and final 30% of layers exhibit higher sensitivities than the remaining layers. PMPQ demonstrates an 11% higher compression ratio than other methods for classification tasks, while TDMPQ achieves a 30% greater compression ratio for language modeling tasks.