A Hardware-Aware, Per-Layer Methodology for Post-Training Quantization of Large Language Models

TL;DR

This paper introduces SOP, a hardware-aware post-training quantization method for large language models that achieves near-lossless fidelity at 4.5--6 bits per weight, reducing storage and improving efficiency.

Contribution

It presents a novel quantization methodology combining codebook search, activation-weighted selection, and hardware-efficient formats, outperforming traditional FP8 baselines.

Findings

Achieves lower weight reconstruction error than FP8 at 1.5 bpw lower storage.

Demonstrates effectiveness across six open model families.

Introduces a new hardware-efficient LUT output format (HIF).

Abstract

Scaled Outer Product (SOP) is a post-training quantization methodology for large language model weights, designed to deliver near-lossless fidelity at 4.5--6 bits per weight on hardware with per-layer LUT decode. The methodology combines per-layer search of fixed and dynamic codebook pairs selected by a per-block selection bit, signed per-block scales, activation-weighted cosine selection, and multiple-choice knapsack promotion of sensitive layers with outlier and sparse-residual correction. Fixed codebooks include NF4, BOF4, Split87, and SH4; per-layer optimized codebooks (DD4) are hosted in LUT SRAM. A new hardware-efficient LUT output format (HIF) is proposed to improve performance, energy, and cost. Across six open model families, the recommended FP6 operating point (E2M3sUE4M4, 6.5 bpw) achieves lower weight reconstruction error than the conventional per-layer-POT FP8 baseline (E4M3, 8.0 bpw) at 1.5 bpw lower storage cost, demonstrating that block-scaled small atoms with carefully chosen scale precision can replace conventionally-deployed FP8. Full evaluation across the 4.5--6 bpw range, including layer promotion and sparse residual correction, is reported in a companion paper.