Dependency-Aware Semi-Structured Sparsity of GLU Variants in Large Language Models

TL;DR

This paper introduces Dependency-aware Semi-structured Sparsity (DaSS), a novel pruning method for GLU-based large language models that balances structural dependency with unstructured pruning to improve hardware efficiency and model performance.

Contribution

DaSS incorporates dependency-aware pruning metrics for GLU models, outperforming existing methods in achieving hardware-friendly sparsity while maintaining efficiency.

Findings

DaSS outperforms SparseGPT and Wanda in hardware efficiency.

DaSS maintains computational efficiency comparable to Wanda.

Empirical results on LLaMA2, Mistral, and Gemma show improved sparsity patterns.

Abstract

The rapid advancement in Large Language Models (LLMs) has markedly enhanced the capabilities of language understanding and generation. However, the substantial model size poses hardware challenges, affecting both memory size for serving and inference latency for token generation. To address those challenges, we propose Dependency-aware Semi-structured Sparsity (DaSS), a novel method for the recent prevalent GLU-based LLMs pruning, which incorporates structural dependency into the weight magnitude-based unstructured pruning. We introduce an MLP-specific pruning metric that evaluates the importance of each weight by jointly considering its magnitude and its corresponding MLP intermediate activation norms. DaSS facilitates a balance between the adaptability offered by unstructured pruning and the structural consistency inherent in dependency-based structured pruning. Empirical evaluations on LLaMA2, Mistral, and Gemma model families demonstrate that DaSS not only outperforms both SparseGPT and Wanda in achieving hardware-friendly N:M sparsity patterns but also maintains the computational efficiency of Wanda.