PEANuT: Parameter-Efficient Adaptation with Weight-aware Neural Tweakers

TL;DR

PEANuT introduces weight-aware neural tweakers for parameter-efficient fine-tuning, enabling more expressive updates than linear methods while reducing computational costs, demonstrated across NLP and vision benchmarks.

Contribution

It proposes a novel PEFT framework with weight-aware modules that enhance expressiveness without full model tuning, outperforming existing methods.

Findings

PEANuT outperforms strong baselines in NLP and vision tasks.

Achieves comparable or greater expressivity with fewer parameters.

Maintains low computational overhead during fine-tuning.

Abstract

Fine-tuning large pre-trained foundation models often yields excellent downstream performance but is prohibitively expensive when updating all parameters. Parameter-efficient fine-tuning (PEFT) methods such as LoRA alleviate this by introducing lightweight update modules, yet they commonly rely on weight-agnostic linear approximations, limiting their expressiveness. In this work, we propose PEANuT, a novel PEFT framework that introduces weight-aware neural tweakers, compact neural modules that generate task-adaptive updates conditioned on frozen pre-trained weights. PEANuT provides a flexible yet efficient way to capture complex update patterns without full model tuning. We theoretically show that PEANuT achieves equivalent or greater expressivity than existing linear PEFT methods with comparable or fewer parameters. Extensive experiments across four benchmarks with over twenty datasets demonstrate that PEANuT consistently outperforms strong baselines in both NLP and vision tasks, while maintaining low computational overhead.