Post-Training Statistical Calibration for Higher Activation Sparsity

TL;DR

SCAP is a post-training activation pruning method that enhances sparsity and speed in large language models by calibrating activation distributions, achieving significant efficiency gains across various Transformer architectures.

Contribution

Introduces a novel post-training calibration framework, SCAP, that generalizes activation sparsification for Transformers and improves decoding speed without retraining.

Findings

Achieves 1.5x speedup over CATS at same model quality.

Effectively applied across diverse Transformer models including MoE and pre-quantized models.

Demonstrates robustness and scalability of the method.

Abstract

We present Statistical Calibrated Activation Pruning (SCAP), a post-training activation pruning framework that (1) generalizes sparsification by input activations of Fully-Connected layers for generic and flexible application across Transformers, and (2) features a simple Mode-Centering technique to pre-calibrate activation distributions for maximizing post-training sparsity. Our results demonstrate robust Pareto efficiency compared to prior methods, translating to a 1.5x additional LLM decoding speedup against CATS at iso model quality. SCAP effectiveness is empirically verified across a wide range of models, including recent Transformer Decoders, MoE, Mamba2, Encoding Transformer, and pre-quantized models, highlighting its practicality and scalability. The code is available at: https://github.com/IntelLabs/SCAP.