Astro: Activation-guided Structured Regularization for Outlier-Robust LLM Post-Training Quantization

TL;DR

Astro is a novel regularization framework that improves post-training quantization of LLMs by suppressing outliers through activation-guided weight reconstruction, achieving high performance with zero inference latency.

Contribution

Astro introduces an activation-guided structured regularization method that effectively suppresses outliers in LLM weights during post-training quantization without adding inference latency.

Findings

Outperforms complex rotation methods in quantization accuracy.

Achieves nearly one-third of the quantization time of existing methods.

Maintains accuracy while eliminating inference latency.

Abstract

Weight-only post-training quantization (PTQ) is crucial for efficient Large Language Model (LLM) deployment but suffers from accuracy degradation caused by weight and activation outliers. Existing mitigation strategies often face critical limitations: they either yield insufficient outlier suppression or incur significant deployment inefficiencies, such as inference latency, heavy preprocessing, or reliance on complex operator fusion. To resolve these limitations, we leverage a key insight: over-parameterized LLMs often converge to Flat Minima, implying a vast equivalent solution space where weights can be adjusted without compromising accuracy. Building on this, we propose Astro, an Activation-guided Structured Regularization framework designed to suppress the negative effects of outliers in a hardware-friendly and efficient manner. Leveraging the activation-guided regularization objective, Astro actively reconstructs intrinsically robust weights, aggressively suppressing weight outliers corresponding to high-magnitude activations without sacrificing model accuracy. Crucially, Astro introduces zero inference latency and is orthogonal to mainstream quantization methods like GPTQ. Extensive experiments show that Astro achieves highly competitive performance; notably, on LLaMA-2-7B, it achieves better performance than complex learning-based rotation methods with almost 1/3 of the quantization time.