PrunePEFT: Iterative Hybrid Pruning for Parameter-Efficient Fine-tuning of LLMs

TL;DR

PrunePEFT introduces an iterative hybrid pruning method that optimizes parameter-efficient fine-tuning configurations for large language models, reducing search overhead and enhancing performance.

Contribution

It formulates PEFT strategy search as a pruning problem, enabling efficient, iterative removal of redundant modules to improve fine-tuning outcomes.

Findings

Reduces computational cost of PEFT configuration search

Improves fine-tuning performance with optimized module selection

Scales effectively to large pre-trained models

Abstract

Parameter Efficient Fine-Tuning (PEFT) methods have emerged as effective and promising approaches for fine-tuning pre-trained language models. Compared with Full parameter Fine-Tuning (FFT), PEFT achieved comparable task performance with a substantial reduction of trainable parameters, which largely saved the training and storage costs. However, using the PEFT method requires considering a vast design space, such as the type of PEFT modules and their insertion layers. Inadequate configurations can lead to sub-optimal results. Conventional solutions such as architectural search techniques, while effective, tend to introduce substantial additional overhead. In this paper, we propose a novel approach, PrunePEFT, which formulates the PEFT strategy search as a pruning problem and introduces a hybrid pruning strategy that capitalizes on the sensitivity of pruning methods to different PEFT modules. This method extends traditional pruning techniques by iteratively removing redundant or conflicting PEFT modules, thereby optimizing the fine-tuned configuration. By efficiently identifying the most relevant modules, our approach significantly reduces the computational burden typically associated with architectural search processes, making it a more scalable and efficient solution for fine-tuning large pre-trained models.