Look Within or Look Beyond? A Theoretical Comparison Between Parameter-Efficient and Full Fine-Tuning

TL;DR

This paper provides a theoretical and empirical comparison between Parameter-Efficient Fine-Tuning (PEFT) and Full Fine-Tuning (FFT), showing PEFT's limitations in complex tasks due to its constrained representational capacity.

Contribution

It offers a theoretical demonstration that PEFT is a strict subset of FFT and provides upper bounds on PEFT's representational ability, validated by extensive experiments.

Findings

PEFT is a strict subset of FFT in representational capacity.

PEFT performs well on simple benchmarks but less so on complex tasks.

Theoretical bounds explain PEFT's limitations in robustness and expressiveness.

Abstract

Parameter-Efficient Fine-Tuning (PEFT) methods achieve performance comparable to Full Fine-Tuning (FFT) while requiring significantly fewer computing resources, making it the go-to choice for researchers. We find that although PEFT can achieve competitive results on some benchmarks, its performance falls short of FFT in complex tasks, such as reasoning and instruction-based fine-tuning. In this paper, we compare the characteristics of PEFT and FFT in terms of representational capacity and robustness based on optimization theory. We theoretically demonstrate that PEFT is a strict subset of FFT. By providing theoretical upper bounds for PEFT, we show that the limited parameter space constrains the model's representational ability, making it more susceptible to perturbations. Experiments on 15 datasets encompassing classification, generation, reasoning, instruction fine-tuning tasks and 11 adversarial test sets validate our theories. We hope that these results spark further research beyond the realms of well established PEFT. The source code is in the anonymous Github repository\footnote{https://github.com/misonsky/PEFTEval}.