Enhancing Post-Training Quantization via Future Activation Awareness

TL;DR

This paper introduces Future-Aware Quantization (FAQ), a novel post-training quantization method that uses future-layer activations to improve model compression accuracy and stability without additional training or significant computational overhead.

Contribution

The paper proposes a new quantization approach leveraging future activations and a window-wise preview mechanism to enhance accuracy and robustness in LLM compression.

Findings

FAQ outperforms prior PTQ methods in experiments.

It requires no backward passes or data reconstruction.

It is suitable for edge deployment due to low overhead.

Abstract

Post-training quantization (PTQ) is a widely used method to compress large language models (LLMs) without fine-tuning. It typically sets quantization hyperparameters (e.g., scaling factors) based on current-layer activations. Although this method is efficient, it suffers from quantization bias and error accumulation, resulting in suboptimal and unstable quantization, especially when the calibration data is biased. To overcome these issues, we propose Future-Aware Quantization (FAQ), which leverages future-layer activations to guide quantization. This allows better identification and preservation of important weights, while reducing sensitivity to calibration noise. We further introduce a window-wise preview mechanism to softly aggregate multiple future-layer activations, mitigating over-reliance on any single layer. To avoid expensive greedy search, we use a pre-searched configuration to minimize overhead. Experiments show that FAQ consistently outperforms prior methods with negligible extra cost, requiring no backward passes, data reconstruction, or tuning, making it well-suited for edge deployment.