GenFT: A Generative Parameter-Efficient Fine-Tuning Method for Pretrained Foundation Models

TL;DR

GenFT introduces a novel parameter-efficient fine-tuning method that leverages pretrained weights to guide task-specific updates, significantly improving adaptation efficiency and performance across vision and language benchmarks.

Contribution

It proposes a structured approach to extract transferable information from pretrained weights to enhance PEFT, combining row/column transformations and layer-sharing strategies.

Findings

Outperforms state-of-the-art PEFT methods on multiple benchmarks.

Effectively extracts and utilizes structural information from pretrained weights.

Achieves superior task adaptation with fewer resources.

Abstract

Pretrained Foundation Models (PFMs) have transformed numerous applications by enabling efficient adaptation to customized tasks. Parameter-Efficient Fine-Tuning (PEFT) has emerged as a resource-efficient alternative to full fine-tuning, especially leveraging reparameterized weights $\Delta W$ to adapt models for downstream tasks. However, a critical yet underexplored question remains: can we utilize well-pretrained weights $W_0$ to guide the update of task-specific $\Delta W$, avoiding inefficient training it from scratch? To end this, we propose Generative Parameter-Efficient Fine-Tuning (GenFT), a novel method that extracts structured, transferable information from $W_0$ for efficient $\Delta W$ training. To extract row and column structure information, GenFT applies row and column transformations to distill essential patterns from $W_0$. A tailored policy further decomposes $\Delta W$ into layer-shared and layer-specific components, balancing information reuse and individualized flexibility. GenFT is simple yet effective, achieving superior performance across CV and NLP tasks. Extensive experiments on VTAB-1K, FGVC, and GLUE benchmarks demonstrate that GenFT outperforms state-of-the-art PEFT methods, offering a new perspective for efficient model adaptation.