Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment

TL;DR

This paper provides a comprehensive review and assessment of parameter-efficient fine-tuning methods for large pretrained language models, highlighting their effectiveness in reducing computational costs while maintaining performance.

Contribution

It systematically summarizes PEFT techniques, discusses their applications, and evaluates their effectiveness through experiments, offering valuable insights for future research and practical use.

Findings

PEFT methods significantly reduce fine-tuning parameters and memory usage.

Experimental results show comparable performance to full fine-tuning.

PEFT techniques are effective across various NLP tasks.

Abstract

With the continuous growth in the number of parameters of transformer-based pretrained language models (PLMs), particularly the emergence of large language models (LLMs) with billions of parameters, many natural language processing (NLP) tasks have demonstrated remarkable success. However, the enormous size and computational demands of these models pose significant challenges for adapting them to specific downstream tasks, especially in environments with limited computational resources. Parameter Efficient Fine-Tuning (PEFT) offers an effective solution by reducing the number of fine-tuning parameters and memory usage while achieving comparable performance to full fine-tuning. The demands for fine-tuning PLMs, especially LLMs, have led to a surge in the development of PEFT methods, as depicted in Fig. 1. In this paper, we present a comprehensive and systematic review of PEFT methods for PLMs. We summarize these PEFT methods, discuss their applications, and outline future directions. Furthermore, we conduct experiments using several representative PEFT methods to better understand their effectiveness in parameter efficiency and memory efficiency. By offering insights into the latest advancements and practical applications, this survey serves as an invaluable resource for researchers and practitioners seeking to navigate the challenges and opportunities presented by PEFT in the context of PLMs.