ConQuR: Corner Aligned Activation Quantization via Optimized Rotations for LLMs

TL;DR

This paper introduces a lightweight, post-training rotation calibration method for quantizing activations in large language models, improving efficiency and performance without extensive training or storage overhead.

Contribution

It proposes a novel orthogonal rotation calibration technique that aligns activations with hypercube corners using a closed-form solution, enabling efficient online calibration.

Findings

Achieves competitive perplexity and reasoning performance on Llama models.

Avoids costly end-to-end training and large offline activation storage.

Provides an efficient closed-form update for rotation calibration.

Abstract

Large language models (LLMs) are costly to deploy due to their large memory footprint and high inference cost. Weight-activation quantization can reduce these costs, but low-bit activation quantization remains difficult because activation outliers induce large quantization error. Recent rotation-based methods address this by applying orthogonal transformations that redistribute activation magnitude across dimensions, but existing approaches either require expensive end-to-end rotation training or rely on stored activation corpora, introducing significant compute or storage overhead. We propose a lightweight post-training rotation calibration method for LLM activation quantization. Our method learns orthogonal rotations that align normalized activations with the corners of an inscribed hypercube, encouraging activation energy to be distributed more evenly across dimensions. This objective admits an efficient closed-form update via the orthogonal Procrustes problem, avoiding gradient-based optimization over the orthogonal group. We further introduce an online calibration procedure that updates rotations as calibration samples are processed, eliminating the need to store activations on disk and allowing rotations to adapt to quantized activation distributions during calibration. Experiments on Llama-2 and Llama-3 models from 3B to 70B parameters show that our method achieves competitive or improved performance across perplexity benchmarks and common sense reasoning tasks while avoiding both costly end-to-end training and large offline activation storage.