MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Elastic LLMs

TL;DR

MoBiQuant introduces a token-sensitive mixture-of-bits quantization method that allows elastic LLM inference with dynamic precision adjustment, improving flexibility and performance without repeated calibration.

Contribution

It proposes a novel Mixture-of-Bits quantization framework that dynamically adjusts weight precision based on token sensitivity for elastic LLM deployment.

Findings

Matches performance of bit-specific PTQ on LLaMA3-8B

Enables smooth precision switching during inference

Improves generalization for token outliers

Abstract

Changing runtime complexity on cloud and edge devices necessitates elastic large language model (LLM) deployment, where an LLM can be inferred with various quantization precisions based on available computational resources. However, it has been observed that the calibration parameters for quantization are typically linked to specific precisions, which presents challenges during elastic-precision calibration and precision switching at runtime. In this work, we attribute the source of varying calibration parameters to the varying token-level sensitivity caused by a precision-dependent outlier migration phenomenon.Motivated by this observation, we propose \texttt{MoBiQuant}, a novel Mixture-of-Bits quantization framework that adjusts weight precision for elastic LLM inference based on token sensitivity. Specifically, we propose the many-in-one recursive residual quantization that can iteratively reconstruct higher-precision weights and the token-aware router to dynamically select the number of residual bit slices. MoBiQuant enables smooth precision switching while improving generalization for the distribution of token outliers. Experimental results demonstrate that MoBiQuant exhibits strong elasticity, enabling it to match the performance of bit-specific calibrated PTQ on LLaMA3-8B without repeated calibration.