Towards Joint Quantization and Token Pruning of Vision-Language Models

TL;DR

This paper introduces QUOTA, a unified framework for low-bit quantization and deterministic token pruning in vision-language models, improving robustness and efficiency during inference.

Contribution

It proposes a collaborative quantization-and-pruning method that unifies low-bit inference and token pruning in a single pipeline, enhancing robustness and efficiency.

Findings

Achieves 95.65% token retention with only 30% tokens used.

Outperforms stage-wise baselines in robustness under low-bit regimes.

Improves inference efficiency and robustness on standard benchmarks.

Abstract

Deploying Vision-Language Models (VLMs) under aggressive low-bit inference remains challenging because inference cost is dominated by the long visual-token prefix during prefill and the growing KV cache during autoregressive decoding. Token pruning and low-bit quantization are complementary for reducing these costs, yet naive stage-wise combinations are often brittle due to a mismatch between quantization calibration and pruning execution. We present a collaborative quantization-and-pruning framework that unifies low-bit inference and deterministic visual-token pruning in a single deployable pipeline. The framework introduces the \textbf{Q}uantization \textbf{U}nified \textbf{O}ffline \textbf{T}oken \textbf{A}llocator (\textbf{QUOTA}), which converts low-bit calibration signals into a layer-wise token allocation schedule and materializes it as a pruning recipe. Token importance is evaluated under deployed W4A4 operators with a quantized KV cache by combining activation magnitude, attention cues, and an explicit low-bit risk signal, enabling consistent budgeted top-$k$ selection. Experiments on standard VLM benchmarks show improved robustness over stage-wise baselines under the same low-bit regime, achieving 95.65\% average retention while retaining only 30\% of visual tokens, compared with about 94.3\% retention for representative stage-wise combinations. The code will be released.