Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning

TL;DR

This paper systematically reviews parameter-efficient fine-tuning methods for large language models, comparing 15 approaches on models up to 11B parameters, and offers practical recommendations and future research directions.

Contribution

It provides a comprehensive taxonomy, extensive experimental comparison, and practical guidelines for PEFT methods in resource-constrained settings.

Findings

Methods struggle in resource-limited scenarios

Hyperparameter tuning impacts performance significantly

Some methods outperform baseline in specific conditions

Abstract

This paper presents a systematic overview of parameter-efficient fine-tuning methods, covering over 50 papers published between early 2019 and mid-2024. These methods aim to address the challenges of fine-tuning large language models by training only a small subset of parameters. We provide a taxonomy that covers a broad range of methods and present a detailed method comparison with a specific focus on real-life efficiency in fine-tuning multibillion-scale language models. We also conduct an extensive head-to-head experimental comparison of 15 diverse PEFT methods, evaluating their performance and efficiency on models up to 11B parameters. Our findings reveal that methods previously shown to surpass a strong LoRA baseline face difficulties in resource-constrained settings, where hyperparameter optimization is limited and the network is fine-tuned only for a few epochs. Finally, we provide a set of practical recommendations for using PEFT methods and outline potential future research directions.