Olica: Efficient Structured Pruning of Large Language Models without Retraining

TL;DR

Olica is a novel pruning framework for large language models that removes the need for retraining by using PCA and linear calibration, significantly reducing computational costs while maintaining model accuracy.

Contribution

It introduces a retraining-free structured pruning method for LLMs using PCA and SVD, improving efficiency and preserving model performance.

Findings

Reduces pruning complexity by a factor of the square of attention heads

Maintains accuracy without retraining across multiple benchmarks

Uses linear calibration to mitigate error accumulation in pruned models

Abstract

Most existing structured pruning methods for Large Language Models (LLMs) require substantial computational and data resources for retraining to reestablish the corrupted correlations, making them prohibitively expensive. To address this, we propose a pruning framework for LLMs called Orthogonal decomposition and Linear Calibration (Olica), which eliminates the need for retraining. A key observation is that the multi-head attention (MHA) layer depends on two types of matrix products. By treating these matrix products as unified entities and applying principal component analysis (PCA), we extract the most important information to compress LLMs without sacrificing accuracy or disrupting their original structure. Consequently, retraining becomes unnecessary. A fast decomposition method is devised, reducing the complexity of PCA by a factor of the square of the number of attention heads. Additionally, to mitigate error accumulation problem caused by pruning the feed-forward network (FFN) layer, we introduce a linear calibration method to reconstruct the residual errors of pruned layers using low-rank matrices. By leveraging singular value decomposition (SVD) on the solution of the least-squares problem, these matrices are obtained without requiring retraining. Extensive experiments show that the proposed Olica is efficient in terms of data usage, GPU memory, and running time, while delivering superior performance across multiple benchmarks.