Preference-aware Influence-function-based Data Selection Method for Efficient Fine-Tuning

TL;DR

PRISM is a data selection method that uses model preferences to weight target examples and improve fine-tuning efficiency for large language models.

Contribution

It introduces a preference-aware influence function approach that enhances data selection by considering the relevance of target examples to the current model.

Findings

PRISM improves fine-tuning efficiency across various model scales.

It enhances safety-oriented supervised fine-tuning repair.

Preference weighting leads to more effective target-behavior guidance.

Abstract

As LLMs continue to scale, improving training efficiency increasingly depends on using data more effectively. Data selection addresses this problem by allocating a limited training budget to samples that best promote a target behavior. Existing methods usually represent the target behavior with a set of target examples, but often treat these examples as equally important. This can be inefficient because target examples may differ in their relevance to the current model: examples closer to the model's current behavior provide more actionable guidance than those farther away. We propose PRISM (PReference-aware Influence-function-based Data Selection Method for Efficient Fine-Tuning), which uses the current model's preference to weight target examples and construct a preference-aware target representation. PRISM then scores candidate training samples by their alignment with this representation, concentrating the data budget on samples more likely to move the model toward the target behavior. Theoretical analysis shows that this preference weighting yields a more effective first-order direction for increasing target-behavior preference. Experiments across model families and scales show that PRISM improves both efficient fine-tuning and safety-oriented SFT repair, demonstrating that precise target-behavior characterization is key to budget-efficient data selection.