SPAFIT: Stratified Progressive Adaptation Fine-tuning for Pre-trained Large Language Models

TL;DR

SPAFIT is a new parameter-efficient fine-tuning method for large language models that localizes linguistic knowledge to specific layers, outperforming existing PEFT methods on multiple tasks with fewer parameters.

Contribution

The paper introduces SPAFIT, a novel PEFT approach that leverages layer-specific linguistic knowledge for more efficient fine-tuning of large language models.

Findings

SPAFIT outperforms other PEFT methods on GLUE tasks.

SPAFIT fine-tunes fewer parameters than existing methods.

SPAFIT reduces computational and storage requirements.

Abstract

Full fine-tuning is a popular approach to adapt Transformer-based pre-trained large language models to a specific downstream task. However, the substantial requirements for computational power and storage have discouraged its widespread use. Moreover, increasing evidence of catastrophic forgetting and overparameterization in the Transformer architecture has motivated researchers to seek more efficient fine-tuning (PEFT) methods. Commonly known parameter-efficient fine-tuning methods like LoRA and BitFit are typically applied across all layers of the model. We propose a PEFT method, called Stratified Progressive Adaptation Fine-tuning (SPAFIT), based on the localization of different types of linguistic knowledge to specific layers of the model. Our experiments, conducted on nine tasks from the GLUE benchmark, show that our proposed SPAFIT method outperforms other PEFT methods while fine-tuning only a fraction of the parameters adjusted by other methods.