Accurate Retraining-free Pruning for Pretrained Encoder-based Language Models

TL;DR

This paper introduces K-prune, a novel retraining-free structured pruning method for pretrained language models that preserves knowledge to significantly improve accuracy at high compression rates.

Contribution

K-prune is a new retraining-free pruning algorithm that maintains model knowledge, reducing accuracy loss during compression of pretrained language models.

Findings

Achieves up to 58.02% higher F1 score compared to existing methods.

Effectively compresses models by 80% without retraining.

Significantly improves accuracy at high compression rates.

Abstract

Given a pretrained encoder-based language model, how can we accurately compress it without retraining? Retraining-free structured pruning algorithms are crucial in pretrained language model compression due to their significantly reduced pruning cost and capability to prune large language models. However, existing retraining-free algorithms encounter severe accuracy degradation, as they fail to handle pruning errors, especially at high compression rates. In this paper, we propose K-prune (Knowledge-preserving pruning), an accurate retraining-free structured pruning algorithm for pretrained encoder-based language models. K-prune focuses on preserving the useful knowledge of the pretrained model to minimize pruning errors through a carefully designed iterative pruning process composed of knowledge measurement, knowledge-preserving mask search, and knowledge-preserving weight-tuning. As a result, K-prune shows significant accuracy improvements up to 58.02%p higher F1 score compared to existing retraining-free pruning algorithms under a high compression rate of 80% on the SQuAD benchmark without any retraining process.