$\rm SP^3$: Enhancing Structured Pruning via PCA Projection

TL;DR

SP3 introduces a PCA-based structured pruning method that effectively reduces model size by compressing the hidden dimension in PLMs, achieving high compression rates while maintaining accuracy.

Contribution

The paper presents a novel PCA projection-based structured pruning technique (SP3) that specifically targets the hidden dimension for more efficient model compression.

Findings

Reduces hidden dimension by 70% in BERTbase

Compresses 94% of BERTbase with over 96% accuracy

Effective on models like OPT and Llama

Abstract

Structured pruning is a widely used technique for reducing the size of pre-trained language models (PLMs), but current methods often overlook the potential of compressing the hidden dimension (d) in PLMs, a dimension critical to model size and efficiency. This paper introduces a novel structured pruning approach, Structured Pruning with PCA Projection (SP3), targeting the effective reduction of d by projecting features into a space defined by principal components before masking. Extensive experiments on benchmarks (GLUE and SQuAD) show that SP3 can reduce d by 70%, compress 94% of the BERTbase model, maintain over 96% accuracy, and outperform other methods that compress d by 6% in accuracy at the same compression ratio. SP3 has also proven effective with other models, including OPT and Llama. Our data and code are available at an anonymous repo.