Dynamic Subset Tuning: Expanding the Operational Range of Parameter-Efficient Training for Large Language Models

TL;DR

This paper introduces Dynamic Subset Tuning, a flexible parameter-efficient training method that dynamically selects model parameters during training, achieving comparable or better performance than existing methods with fewer parameters.

Contribution

It presents a novel dynamic parameter selection approach for PET that adapts the subset of trainable parameters over time, unlike fixed methods like prompt tuning and LoRA.

Findings

Matches or outperforms prompt tuning and LoRA on NLP tasks

Works across various model sizes and families

Enables scalable subset size adjustment

Abstract

We propose a novel parameter-efficient training (PET) method for large language models that adapts models to downstream tasks by optimizing a small subset of the existing model parameters. Unlike prior methods, this subset is not fixed in location but rather which parameters are modified evolves over the course of training. This dynamic parameter selection can yield good performance with many fewer parameters than extant methods. Our method enables a seamless scaling of the subset size across an arbitrary proportion of the total model size, while popular PET approaches like prompt tuning and LoRA cover only a small part of this spectrum. We match or outperform prompt tuning and LoRA in most cases on a variety of NLP tasks (MT, QA, GSM8K, SuperGLUE) for a given parameter budget across different model families and sizes.