FlexQuant: A Flexible and Efficient Dynamic Precision Switching Framework for LLM Quantization

TL;DR

FlexQuant is a dynamic, layer-wise mixed-precision quantization framework for LLMs that adaptively switches precision during inference, significantly improving speed with minimal accuracy loss.

Contribution

It introduces a novel dynamic precision-switching framework for LLM quantization that adjusts bit-widths during inference based on model perplexity and divergence metrics.

Findings

Achieves 1.3x speedup in inference

Maintains negligible accuracy loss

Provides a comprehensive analysis of quantization strategies

Abstract

The rapid advancement of large language models (LLMs) has exacerbated the memory bottleneck due to the widening gap between model parameter scaling and hardware capabilities. While post-training quantization techniques effectively reduce memory overhead, existing methods predominantly rely on static quantization strategies, which struggle to adapt to dynamic workloads. To address this, we propose FlexQuant, a dynamic precision-switching framework that optimizes the trade-off between inference speed and accuracy. Leveraging model perplexity entropy and Kullback-Leibler divergence, FlexQuant enables fine-grained, layer-wise mixed-precision quantization and dynamically adjusts bit-widths during each token generation. FlexQuant provides a comprehensive analysis of quantization strategies, introduces a precision requirement model for optimal switching, and implements efficient fine-grained precision management. Evaluations demonstrate that FlexQuant achieves a 1.3x end-to-end speedup across diverse language tasks with negligible accuracy loss introduced. This framework offers a flexible and adaptive solution for efficient LLM deployment. Code is released at https://github.com/ZongwuWang/FlexQuant.git.