Adaptive parameter-efficient fine-tuning via Hessian-informed subset selection

TL;DR

This paper introduces AdaPEFT, a Hessian-informed subset selection method for parameter-efficient fine-tuning that optimally balances performance and the number of trainable parameters across diverse tasks and models.

Contribution

It formulates subset selection as a multi-task optimization problem and proposes AdaPEFT, a novel Hessian-informed approach for effective and transferable parameter subset selection in PEFT.

Findings

AdaPEFT outperforms existing PEFT methods in various tasks.

Selected subsets transfer well across training horizons and model sizes.

The method effectively balances performance and parameter efficiency.

Abstract

Parameter-efficient fine-tuning (PEFT) is a highly effective approach for adapting large pre-trained models to downstream tasks with minimal computational overhead. At the core, PEFT methods freeze most parameters and only trains a small subset (say $<0.1\%$ of total parameters). Notably, different PEFT methods select different subsets, resulting in varying levels of performance. This variation prompts a key question: how to effectively select the most influential subset to train? We formulate the subset selection as a multi-task problem: maximizing the performance and minimizing the number of trainable parameters. We leverage a series of transformations -- including $\epsilon$-constraint method and second-order Taylor approximation -- to arrive at the classical 0-1 knapsack problem, which we solve through the lens of Pareto optimality. Consequently, we propose AdaPEFT, a Hessian-informed PEFT that adapts to various tasks and models, in which the selected subset empirically transfers across training horizons and model sizes.