Exploring Layer-wise Information Effectiveness for Post-Training Quantization in Small Language Models

TL;DR

This paper introduces LieQ, a layer-wise, geometry-driven post-training quantization method for small language models that maintains accuracy at ultra-low bit-widths while preserving hardware efficiency.

Contribution

LieQ presents a novel, metric-driven quantization framework that automatically allocates bit-widths based on layer importance, improving accuracy and efficiency for models under 8B parameters.

Findings

LieQ reduces accuracy gap at 2-bit quantization on Qwen3 and LLaMA3.x models.

It preserves standard kernel operations, enabling efficient deployment on edge devices.

Layer-wise functional saliency correlates with representational compactness, guiding bit-width allocation.

Abstract

Large language models with billions of parameters are often over-provisioned: many layers contribute little unique information yet dominate the memory and energy footprint during inference. We present LieQ Layer-wise information effectiveness Quantization, a hardware-native, metric-driven post-training quantization framework that addresses the critical challenge of maintaining accuracy in sub-8B models, model parameters less than 8B, under extreme low-bit compression. LieQ keeps uniform bit-width within each layer while mixing precision across layers, preserving standard multiplication kernels and avoiding irregular memory access, codebooks, or irregular formats at inference time. Our method uncovers a strong correlation between layer-wise functional saliency and representational compactness, revealing that layers with higher training-induced energy concentration are functionally irreplaceable. Leveraging this insight, we propose a purely geometry-driven sensitivity proxy that enables automatic bit-width allocation under a target average-bit budget without expensive gradient updates or inference-based perplexity probing. At sub 2-bit, LieQ consistently reduces the large accuracy gap typically observed for naive 2-bit baselines on Qwen3 and LLaMA3.x families, while retaining standard-kernel efficiency. These properties make LieQ a practical path toward deploying small language models on resource-constrained edge devices. Code will available here: https://github.com/HeXiao-55/LieQ-official.git.