CoQuant: Joint Weight-Activation Subspace Projection for Mixed-Precision LLMs

TL;DR

CoQuant introduces a joint weight-activation subspace projection method for mixed-precision quantization of LLMs, improving inference efficiency while maintaining accuracy by modeling output error with both weight and activation noise.

Contribution

It presents a theoretically grounded closed-form weighted PCA approach that optimally balances weight and activation covariances for better low-bit LLM quantization.

Findings

Outperforms existing PTQ methods on Llama-3.2 and Qwen2.5 models.

Achieves lower perplexity and higher reasoning accuracy.

Provides a principled approach for joint weight-activation subspace modeling.

Abstract

Post-training quantization (PTQ) has become an important technique for reducing the inference cost of Large Language Models (LLMs). While recent mixed-precision methods improve ultra-low bit quantization by preserving critical subspaces in high precision, they typically construct these subspaces relying solely on activation statistics. This ignores the fundamental nature of linear operations, where the output perturbation is jointly driven by both activation and weight quantization noise. In this paper, we propose CoQuant, a joint weight-activation subspace projection method. By theoretically modeling the expected output error, CoQuant formulates a closed-form weighted PCA solution that balances activation and weight covariances to select the optimal high-precision subspace. Extensive experiments on Llama-3.2 and Qwen2.5 models show that CoQuant consistently outperforms strong PTQ baselines in both WikiText perplexity and zero-shot common-sense reasoning accuracy. These results demonstrate that joint weight-activation subspace modeling provides a principled and effective direction for low-bit LLM quantization. The source code is available at https://github.com/Zachary5895/CoQuant.