Iterative Structured Pruning for Large Language Models with Multi-Domain Calibration

TL;DR

This paper proposes an iterative structured pruning method for large language models that uses multi-domain calibration to effectively reduce model size while maintaining performance, facilitating deployment on standard hardware.

Contribution

It introduces a novel structured pruning framework with a hybrid multi-domain calibration and iterative strategy, improving compression efficiency for LLMs.

Findings

Achieves significant model compression with minimal accuracy loss.

Effective removal of redundant channels across multiple tasks.

Compatible with standard hardware accelerators.

Abstract

Large Language Models (LLMs) have achieved remarkable success across a wide spectrum of natural language processing tasks. However, their ever-growing scale introduces significant barriers to real-world deployment, including substantial computational overhead, memory footprint, and inference latency. While model pruning presents a viable solution to these challenges, existing unstructured pruning techniques often yield irregular sparsity patterns that necessitate specialized hardware or software support. In this work, we explore structured pruning, which eliminates entire architectural components and maintains compatibility with standard hardware accelerators. We introduce a novel structured pruning framework that leverages a hybrid multi-domain calibration set and an iterative calibration strategy to effectively identify and remove redundant channels. Extensive experiments on various models across diverse downstream tasks show that our approach achieves significant compression with minimal performance degradation.