MI-PRUN: Optimize Large Language Model Pruning via Mutual Information

TL;DR

This paper introduces MI-PRUN, a mutual information-based method for large language model pruning that improves stability and achieves near-optimal compression by evaluating hidden state transitions and using an efficient block selection algorithm.

Contribution

We propose MI-PRUN, a novel pruning approach leveraging mutual information and the Data Processing Inequality to identify redundant blocks in LLMs, with an efficient algorithm for near-global optimality.

Findings

Demonstrates improved stability over existing pruning methods.

Achieves significant model compression and acceleration.

Proves effectiveness across various models and datasets.

Abstract

Large Language Models (LLMs) have become indispensable across various domains, but this comes at the cost of substantial computational and memory resources. Model pruning addresses this by removing redundant components from models. In particular, block pruning can achieve significant compression and inference acceleration. However, existing block pruning methods are often unstable and struggle to attain globally optimal solutions. In this paper, we propose a mutual information based pruning method MI-PRUN for LLMs. Specifically, we leverages mutual information to identify redundant blocks by evaluating transitions in hidden states. Additionally, we incorporate the Data Processing Inequality (DPI) to reveal the relationship between the importance of entire contiguous blocks and that of individual blocks. Moreover, we develop the Fast-Block-Select algorithm, which iteratively updates block combinations to achieve a globally optimal solution while significantly improving the efficiency. Extensive experiments across various models and datasets demonstrate the stability and effectiveness of our method.