Restoring Pruned Large Language Models via Lost Component Compensation

TL;DR

This paper introduces RestoreLCC, a novel method for restoring pruned large language models by selectively reintroducing lost information in attention heads, significantly improving performance without sacrificing efficiency.

Contribution

It proposes a targeted, plug-and-play restoration technique that leverages attention activation differences to recover pruned model performance more effectively than existing methods.

Findings

RestoreLCC outperforms state-of-the-art baselines in performance recovery.

The method is compatible with various pruning schemes.

RestoreLCC maintains model sparsity and inference efficiency.

Abstract

Pruning is a widely used technique to reduce the size and inference cost of large language models (LLMs), but it often causes performance degradation. To mitigate this, existing restoration methods typically employ parameter-efficient fine-tuning (PEFT), such as LoRA, to recover the pruned model's performance. However, most PEFT methods are designed for dense models and overlook the distinct properties of pruned models, often resulting in suboptimal recovery. In this work, we propose a targeted restoration strategy for pruned models that restores performance while preserving their low cost and high efficiency. We observe that pruning-induced information loss is reflected in attention activations, and selectively reintroducing components of this information can significantly recover model performance. Based on this insight, we introduce RestoreLCC (Restoring Pruned LLMs via Lost Component Compensation), a plug-and-play method that contrastively probes critical attention heads via activation editing, extracts lost components from activation differences, and finally injects them back into the corresponding pruned heads for compensation and recovery. RestoreLCC is compatible with structured, semi-structured, and unstructured pruning schemes. Extensive experiments demonstrate that RestoreLCC consistently outperforms state-of-the-art baselines in both general and task-specific performance recovery, without compromising the sparsity or inference efficiency of pruned models.