FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain

TL;DR

FisherSFT introduces an information gain-based method for selecting training data to improve the efficiency of supervised fine-tuning of large language models, reducing data requirements while maintaining performance.

Contribution

The paper proposes a novel data selection technique using Hessian-based information gain to enhance the statistical efficiency of supervised fine-tuning of LLMs.

Findings

Significant reduction in training data needed for effective fine-tuning.

Efficient approximation of information gain via linearization at the last layer.

Empirical results show improved performance with fewer training examples.

Abstract

Supervised fine-tuning (SFT) is a standard approach to adapting large language models (LLMs) to new domains. In this work, we improve the statistical efficiency of SFT by selecting an informative subset of training examples. Specifically, for a fixed budget of training examples, which determines the computational cost of fine-tuning, we determine the most informative ones. The key idea in our method is to select examples that maximize information gain, measured by the Hessian of the log-likelihood of the LLM. We approximate it efficiently by linearizing the LLM at the last layer using multinomial logistic regression models. Our approach is computationally efficient, analyzable, and performs well empirically. We demonstrate this on several problems, and back our claims with both quantitative results and an LLM evaluation.