Data Valuation for LLM Fine-Tuning: Efficient Shapley Value Approximation via Language Model Arithmetic

TL;DR

This paper introduces a scalable method for data valuation in large language models by leveraging the mathematical structure of Direct Preference Optimization, enabling efficient Shapley value approximation for better data curation and collaboration.

Contribution

The paper presents a novel approach that simplifies Shapley value computation for LLMs trained with DPO, facilitating practical data valuation.

Findings

Shapley values can be efficiently approximated for DPO-trained LLMs.

The method reduces computational costs significantly compared to traditional approaches.

Enables fair data sharing and investment decisions among multiple data owners.

Abstract

Data is a critical asset for training large language models (LLMs), alongside compute resources and skilled workers. While some training data is publicly available, substantial investment is required to generate proprietary datasets, such as human preference annotations or to curate new ones from existing sources. As larger datasets generally yield better model performance, two natural questions arise. First, how can data owners make informed decisions about curation strategies and data sources investment? Second, how can multiple data owners collaboratively pool their resources to train superior models while fairly distributing the benefits? This problem, data valuation, which is not specific to large language models, has been addressed by the machine learning community through the lens of cooperative game theory, with the Shapley value being the prevalent solution concept. However, computing Shapley values is notoriously expensive for data valuation, typically requiring numerous model retrainings, which can become prohibitive for large machine learning models. In this work, we demonstrate that this computational challenge is dramatically simplified for LLMs trained with Direct Preference Optimization (DPO). We show how the specific mathematical structure of DPO enables scalable Shapley value computation. We believe this observation unlocks many applications at the intersection of data valuation and large language models.