Beyond Outliers: A Data-Free Layer-wise Mixed-Precision Quantization Approach Driven by Numerical and Structural Dual-Sensitivity

TL;DR

This paper introduces NSDS, a novel layer-wise mixed-precision quantization method that considers both numerical and structural sensitivities of neural network layers, enabling more effective compression without calibration data.

Contribution

The paper proposes a dual-sensitivity framework for mixed-precision quantization that decomposes layers into operational roles and aggregates sensitivity scores for improved bit allocation.

Findings

NSDS outperforms existing methods across multiple models and tasks.

It achieves higher accuracy with lower bit-widths.

No calibration data is required for its operation.

Abstract

Layer-wise mixed-precision quantization (LMPQ) enables effective compression under extreme low-bit settings by allocating higher precision to sensitive layers. However, existing methods typically treat all intra-layer weight modules uniformly and rely on a single numerical property when estimating sensitivity, overlooking their distinct operational roles and structural characteristics. To address this, we propose NSDS, a novel calibration-free LMPQ framework driven by Numerical and Structural Dual-Sensitivity. Specifically, it first mechanistically decomposes each layer into distinct operational roles and quantifies their sensitivity from both numerical and structural perspectives. These dual-aspect scores are then aggregated into a unified layer-wise metric through a robust aggregation scheme based on MAD-Sigmoid and Soft-OR to guide bit allocation. Extensive experiments demonstrate that NSDS consistently achieves superior performance compared to various baselines across diverse models and downstream tasks, without relying on any calibration data.