S2D: Selective Spectral Decay for Quantization-Friendly Conditioning of Neural Activations

TL;DR

This paper introduces S2D, a spectral decay method that regularizes neural network weights to reduce activation outliers, thereby improving quantization accuracy and enabling efficient deployment of large-scale models.

Contribution

The paper presents a novel spectral decay regularization technique that targets dominant singular values to improve quantization friendliness of neural activations.

Findings

S2D reduces activation outliers and improves PTQ accuracy by up to 7%.

Models trained with S2D show better quantization robustness across tasks.

S2D enables scaling large models without sacrificing deployment efficiency.

Abstract

Activation outliers in large-scale transformer models pose a fundamental challenge to model quantization, creating excessively large ranges that cause severe accuracy drops during quantization. We empirically observe that outlier severity intensifies with pre-training scale (e.g., progressing from CLIP to the more extensively trained SigLIP and SigLIP2). Through theoretical analysis as well as empirical correlation studies, we establish the direct link between these activation outliers and dominant singular values of the weights. Building on this insight, we propose Selective Spectral Decay ($S^2D$), a geometrically-principled conditioning method that surgically regularizes only the weight components corresponding to the largest singular values during fine-tuning. Through extensive experiments, we demonstrate that $S^2D$ significantly reduces activation outliers and produces well-conditioned representations that are inherently quantization-friendly. Models trained with $S^2D$ achieve up to 7% improved PTQ accuracy on ImageNet under W4A4 quantization and 4% gains when combined with QAT. These improvements also generalize across downstream tasks and vision-language models, enabling the scaling of increasingly large and rigorously trained models without sacrificing deployment efficiency.