Beyond One-Size-Fits-All Pruning via Evolutionary Metric Search for Large Language Models

TL;DR

This paper introduces extsc{OptiShear}, an evolutionary framework for adaptive pruning of large language models that accounts for diverse weight distributions, improving compression efficiency and generalizability across models and tasks.

Contribution

The paper presents a novel evolutionary optimization framework with a new Meta pruning metric and model-wise reconstruction error for adaptive LLM pruning, outperforming existing methods.

Findings

Adaptive pruning metrics outperform fixed strategies.

Layerwise sparsity ratios improve pruning effectiveness.

Framework generalizes across models and tasks.

Abstract

Post-training pruning has emerged as a crucial optimization technique as large language models (LLMs) continue to grow rapidly. However, the significant variations in weight distributions across different LLMs make fixed pruning strategies inadequate for multiple models. In this paper, we introduce \textbf{\textsc{OptiShear}}, an efficient evolutionary optimization framework for adaptive LLM pruning. Our framework features two key innovations: an effective search space built on our Meta pruning metric to handle diverse weight distributions, and a model-wise reconstruction error for rapid evaluation during search trials. We employ Non-dominated Sorting Genetic Algorithm III (NSGA-III) to optimize both pruning metrics and layerwise sparsity ratios. Through extensive evaluation on LLaMA-1/2/3 and Mistral models (7B-70B) across multiple benchmarks, we demonstrate that our adaptive pruning metrics consistently outperform existing methods. Additionally, our discovered layerwise sparsity ratios enhance the effectiveness of other pruning metrics. The framework exhibits strong cross-task and cross-model generalizability, providing a cost-effective solution for model compression.