Scalable Fine-tuning from Multiple Data Sources: A First-Order Approximation Approach

TL;DR

This paper presents a fast, gradient-based method for selecting optimal auxiliary tasks to improve language model fine-tuning, significantly reducing computation time while maintaining high accuracy.

Contribution

Introduces a novel first-order approximation algorithm for efficient subset selection in LM fine-tuning, avoiding repeated training on task subsets.

Findings

Achieves 30x speedup over traditional subset selection methods.

Maintains only 1% error in estimating true fine-tuning performance.

Improves downstream task performance by up to 3.8% over existing methods.

Abstract

We study the problem of fine-tuning a language model (LM) for a target task by optimally using the information from $n$ auxiliary tasks. This problem has broad applications in NLP, such as targeted instruction tuning and data selection in chain-of-thought fine-tuning. The key challenge of this problem is that not all auxiliary tasks are beneficial in improving the performance of the target task. Thus, selecting the right subset of auxiliary tasks is crucial. Conventional subset selection methods, such as forward and backward stepwise selection, are unsuitable for LM fine-tuning because they require repeated training on subsets of auxiliary tasks. This paper introduces a new algorithm for estimating model fine-tuning performance without requiring repeated training. Our algorithm first performs multitask training using data from all tasks to obtain a meta initialization. Then, we approximate the model fine-tuning loss of a subset using functional values and gradients from the meta initialization. Empirically, we find that this gradient-based approximation holds with remarkable accuracy for twelve transformer-based LMs. Thus, we can now estimate fine-tuning performances on CPUs within a few seconds. Finally, we fine-tune the pretrained base model once on the selected subset of tasks. We conduct extensive experiments to validate this approach, delivering a speedup of $30\times$ over conventional subset selection while incurring only $1\%$ error of the true fine-tuning performances. In downstream evaluations involving both instruction tuning and chain-of-thought fine-tuning, this loss-based selection approach improves over prior gradient or representation similarity-based methods for subset selection by up to $3.8\%$.