BIPEFT: Budget-Guided Iterative Search for Parameter Efficient Fine-Tuning of Large Pretrained Language Models

TL;DR

BIPEFT introduces a novel, efficient iterative search method for automatic parameter-efficient fine-tuning of large language models, improving performance and speed by intelligently managing parameter budgets.

Contribution

It proposes a new iterative search strategy that disentangles search spaces and integrates parameter budgets, significantly enhancing automatic PEFT efficiency and effectiveness.

Findings

Outperforms existing PEFT methods on benchmark tasks.

Achieves comparable or better performance with fewer parameters.

Speeds up the fine-tuning process through early selection strategies.

Abstract

Parameter Efficient Fine-Tuning (PEFT) offers an efficient solution for fine-tuning large pretrained language models for downstream tasks. However, most PEFT strategies are manually designed, often resulting in suboptimal performance. Recent automatic PEFT approaches aim to address this but face challenges such as search space entanglement, inefficiency, and lack of integration between parameter budgets and search processes. To overcome these issues, we introduce a novel Budget-guided Iterative search strategy for automatic PEFT (BIPEFT), significantly enhancing search efficiency. BIPEFT employs a new iterative search strategy to disentangle the binary module and rank dimension search spaces. Additionally, we design early selection strategies based on parameter budgets, accelerating the learning process by gradually removing unimportant modules and fixing rank dimensions. Extensive experiments on public benchmarks demonstrate the superior performance of BIPEFT in achieving efficient and effective PEFT for downstream tasks with a low parameter budget.